<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Huey Fern Tay on Medium]]></title>
        <description><![CDATA[Stories by Huey Fern Tay on Medium]]></description>
        <link>https://medium.com/@hftdata?source=rss-137b395910dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3nCOLHVhVHcHfYzRJYgAuQ.jpeg</url>
            <title>Stories by Huey Fern Tay on Medium</title>
            <link>https://medium.com/@hftdata?source=rss-137b395910dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:10:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hftdata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Detecting multicollinearity — it’s not that easy sometimes]]></title>
            <link>https://medium.com/codinginr/detecting-multicollinearity-its-not-that-easy-sometimes-98a335140549?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/98a335140549</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[multicollinearity]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[multiple-linearregression]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Wed, 12 Jan 2022 10:39:39 GMT</pubDate>
            <atom:updated>2022-01-12T10:49:39.438Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Detecting multicollinearity — it’s not that easy sometimes</strong></h3><p>By Huey Fern Tay</p><p>With Greg Page</p><p>When are two variables <em>too</em> related to one another to be used together in a linear regression model? Should the maximum acceptable correlation be 0.7? Or should the rule of thumb be 0.8? There is actually no single, ‘one-size-fits-all’ answer to this question.</p><p>As an alternative to using pairwise correlations, a modeler can examine the variance inflation factor, or VIF, associated with each of the numeric input variables. Sometimes, however, pairwise correlations, and even VIF scores, do not tell the entire picture.</p><p>Consider this correlation matrix created from a Los Angeles Airbnb dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*TxqNxHkm9SU3qaX9SpjgIw.png" /></figure><p>Two item pairs identified in the correlation matrix above have a strong correlation value:</p><p>· Beds + log_accommodates (r= 0.701)</p><p>· Beds + bedrooms (r= 0.706)</p><p>Based on one school of thought, these correlation values are cause for concern; meanwhile, other sources suggest that such values are nothing to be worried about.</p><p>The variance inflation factor, which is used to detect the severity of multicollinearity, does not suggest anything unusual either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*Zw_8q-jfeucAVEBmH-a4-w.png" /></figure><p>The VIF for each potential input variable is found by making a separate linear regression model, in which the variable being scored serves as the outcome, and the other numeric variables are the predictors. The VIF score for each variable is found by applying this formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*rxQRA_b6QxsKnlslrTFkuw.png" /></figure><p>When the other numeric inputs explain a large percentage of the variability in some other variable, then that variable will have a high VIF. Some sources will tell you that any VIF above 5 means that a variable should not be used in a model, whereas other sources will say that VIF values below 10 are acceptable. None of the vif() results here appear to be problematic, based on standard cutoff thresholds.</p><p>Based on the vif() results shown above, plus some algebraic manipulation of the VIF formula, we can know that a model that predicts <em>beds</em> as the outcome variable, using <em>log_accommodates</em>, <em>bedrooms</em>, and <em>bathrooms </em>as the inputs, has an r-squared of just a little higher than 0.61. That is verified with the model shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*fvN2QMl0q5WjAV_wPA0DEQ.png" /></figure><p>But look at what happens when we build a multi-linear regression model predicting the price of an Airbnb property listing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*6HAzsB5XlMzyKlHed6rl5A.png" /></figure><p>The model summary hints at a problem because the coefficient for beds is negative. The proper interpretation for each coefficient in this linear model is the way that <em>log_price</em> will be impacted by a one-unit increase in that input, with all other inputs held constant.</p><p>Literally, then, this output indicates that having more beds within a house or apartment will make its rental value go down, when all else is held constant. That not only defies our common sense, but it also contradicts something that we already know to be the case — that bed number and <em>log_price</em> are positively associated. Indeed, the correlation matrix shown above indicates a moderately-strong linear relationship between these values, of 0.4816.</p><p>After dropping <em>‘beds’</em> from the original model, the adjusted R-squared declines only marginally, from 0.4878 to 0.4782.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*xVMYOgn_jHzpf6qDzsc28Q.png" /></figure><p>This tiny decline in adjusted r-squared is not worrisome at all. The very low p-value associated with this model’s F-statistic indicates a highly significant overall model. Moreover, the signs of the coefficients for each of these inputs are consistent with the directionality that we see in the original correlation matrix.</p><p>Moreover, we still need to include other important variables that determine real estate pricing e.g. location and property type. After factoring in these categories along with other considerations such as pool availability, cleaning fee, and pet-friendly options, the model’s adjusted R-squared value is pushed to 0.6694. In addition, the residual standard error declines from 0.5276 in the original model to 0.4239.</p><p>Long story short: we cannot be completely reliant on rules of thumb, or even cutoff thresholds from textbooks, when evaluating the multicollinearity risk associated with specific numeric inputs. We must also examine model coefficients’ signs. When a coefficient’s sign “flips” from the direction that we should expect, based on that variable’s correlation with the response variable, that can also indicate that our model coefficients are impacted by multicollinearity.</p><p><em>Data source:</em><a href="http://insideairbnb.com/los-angeles/"><em> Inside Airbnb</em></a></p><p><em>Helpful resources: </em><a href="http://www.r-bloggers.com/"><em>R Bloggers</em></a><em> and </em><a href="https://r4ds.had.co.nz/introduction.html"><em>R for Data Science</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a335140549" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codinginr/detecting-multicollinearity-its-not-that-easy-sometimes-98a335140549">Detecting multicollinearity — it’s not that easy sometimes</a> was originally published in <a href="https://medium.com/codinginr">codinginr</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to analyze the NAs in multiple response surveys]]></title>
            <link>https://medium.com/codex/how-to-analyse-multiple-response-surveys-7dde4c6feb9d?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dde4c6feb9d</guid>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[multiple-response-surveys]]></category>
            <category><![CDATA[survey-analysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 03:29:53 GMT</pubDate>
            <atom:updated>2022-01-12T09:36:28.868Z</atom:updated>
            <content:encoded><![CDATA[<h4>By Huey Fern Tay, With Greg Page</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gQS5iRFk6B9d3DBi" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Most likely, you have encountered questionnaires which asked you to <em>‘select-all-that-apply’</em> in response to some particular prompt. These multiple response questions have their drawbacks, but they are an efficient way of condensing several possible answers into one question on a single page.</p><p>An important part of analysing survey questions involves calculating the non-response rates. When non-response rates spike for a particular question, that might suggest that respondents did not wish to answer it (maybe a question was too personal), that an unclear question needs rewording, or perhaps that a survey is simply too long.</p><p>When survey response data is stored in a CSV, each segment of a multiple response question typically receives its own column — and from the standpoint of organizing the data in a format conducive to quick analysis, that makes sense.</p><p>Suppose we were interested in the responses to Question 7 (<em>“what programming languages do you use on a regular basis?” (Select all that apply))</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*bArNEa8QWrHLTVMMsW_JpA.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>The complete breakdown of answers to all parts of this question is depicted in tabular form, in the screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*9DrSwA1Yil2zG9dNJXW4ug.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>However, determining the true extent of the missing values for multiple response questions is not as simple as calculating the average number of null values. Such a process, demonstrated in the code below, would show a misleading high number for the multiple response questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*X84zuyR2TNJIkDewMy475A.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>Chart 1, shown below, delivers a graphical representation of the per-question missing value averages.</p><p><strong>Chart 1</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*TD83SS3PKgjwB7RqelmeLg.png" /></figure><p>This chart does not distinguish between single response and multiple response questions; instead, it lumps all questions together, registers a respondent’s answer as NaN whenever any part of the question goes unanswered, and then spits out the mean value of those NaNs per question.</p><p>Chart 1 would mislead us into thinking that the non-response rate to Question 7 was greater than 80%, when it was approximately 10.62%. (based on the percentage of respondents who didn’t answer any option at all)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*fCmiNNiLnvm0jrQweSWGYA.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>A better approach for capturing the response rate patterns is to separate the single response questions from the multiple response questions.</p><p><strong>Chart 2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*PY-snOuET9TTvWH9VPC2Yw.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>As you can see from Chart 2, most people did not ignore Question 7. In fact, 15531 people said Python was the programming language they used the most often, a response that represented 77% of all respondents. But because not as many people went on to select other options like Bash, C++, and Javascript, that increased the<em> average</em> non-response rate for this question.</p><p>What else does Chart 2 tell us about missing values in the questionnaire? Let’s examine it together with Chart 3, which shows the extent to which people answered single response questions such as: <em>“What is your age”, “For how many years have you been programming / writing code?”</em></p><p><strong>Chart 3:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*uqn2S9qOZCbAWhMjL3CAdA.png" /><figcaption>Above: Author’s own image</figcaption></figure><p>We see that response counts drop to 50% at around Question 25. The downward trend continues until the uptick towards the very end (Question 38 and 39).</p><p>What can we infer from these results?</p><p>By giving people the option of selecting from a range of alternatives including ‘none’, the survey designers reduced the probability of social desirability bias, which happens when people answer questions in a way that makes them look good according to social norms.</p><p>Questions that ask respondents to <em>‘select-all-that-apply’</em> also shorten the time it takes to complete the survey, because all options are presented in one question, on a single page. In contrast, even though respondents may be compelled to think a little deeper for each single-response question, they may quit the survey prematurely because they will need more time to complete the survey. Imagine what would happen if we broke down all the multiple response questions into something like this?</p><p><strong>Do you use Python?</strong></p><p>Yes</p><p>No</p><p><strong>Do you use SQL?</strong></p><p>Yes</p><p>No</p><p><strong>Do you use R?</strong></p><p>Yes <br>No</p><p>Repeat this format 9 more times for just one question. The survey would be far too tedious to complete.</p><p>That said, the survey was probably a little long for most people. All multiple response questions contained a ‘none’ option, which means respondents could have checked that box if the question was not relevant to them. Since people did not select that option in many instances, we can assume they skipped those questions.</p><p>At the same time, useful insights can be gleaned from the folks who completed all 39 questions. These people are enthusiastic individuals who genuinely want to help the survey designers understand developments within the data science community. The survey designers could consider conducting extensive qualitative interviews with some of these respondents to obtain a deeper insight into industry developments.</p><p><em>Data source: </em><a href="https://www.kaggle.com/c/kaggle-survey-2020"><em>Kaggle</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dde4c6feb9d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/how-to-analyse-multiple-response-surveys-7dde4c6feb9d">How to analyze the NAs in multiple response surveys</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 things I learned by analysing Kaggle’s 2020 Machine Learning and Data Science Survey]]></title>
            <link>https://hftdata.medium.com/5-things-i-learned-by-analysing-kaggles-2020-machine-learning-and-data-science-survey-63311fbe4345?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/63311fbe4345</guid>
            <category><![CDATA[seaborn]]></category>
            <category><![CDATA[conditional-probability]]></category>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[survey-analysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Thu, 18 Nov 2021 09:30:03 GMT</pubDate>
            <atom:updated>2021-11-18T09:30:03.780Z</atom:updated>
            <content:encoded><![CDATA[<p>By Huey Fern Tay</p><p>With Greg Page</p><p>Every year since 2017, Kaggle has canvassed the opinions of its email subscribers, discussion forum members, and social media followers through a large-scale survey aimed at understanding aggregate trends in the world of data analysis and modeling. Survey respondents typically include students as well as business analysts, data scientists, data analysts, data engineers etc. By analysing responses to questions about topics such as the machine learning tools that practitioners are currently using, the tools they hope to become more familiar with, and their recommendations to aspiring data scientists, we can glean interesting insights about major trends in the data science community.</p><p>In the interest of brevity, this article examines just a few aspects of the platform’s 2020 survey.</p><ol><li><strong>Python reigns supreme regardless of occupation</strong></li></ol><p>Python was the runaway winner, with 15,530 respondents saying they used it on a regular basis. That figure represents 80% of the sample size for this question. The heat map below highlights Python’s ubiquity among all professions included in this survey, a prevalence which underscores the language’s versatility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*Zrgx-QT9qvbv1dymYmv9iw.png" /><figcaption>Above: author’s image</figcaption></figure><p>2. <strong>People were nearly 2x more likely to use Python + SQL regularly compared to Python + R</strong></p><p>When we took a deeper look at the question posed above <em>(‘Which programming languages do you use on a regular basis? Select all that apply),</em> it became clear that Python and SQL are a popular combination — 6667 people said they often use both. While it is possible to run SQL queries in R, the R + SQL pairing was approximately half as frequent among survey respondents as was the Python + SQL pairing (22.7% vs 42.9%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*3wcxEG_w5tlKh75dUIGFWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*Xoe5sFWc4iNIn5Y3MZI9QQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*t9Ln-D0FtwLoB3u9eO1KaA.png" /><figcaption>Above: Author’s image</figcaption></figure><p><strong>3.</strong> <strong>Learn Python</strong></p><p>Given Python’s overwhelming popularity among the survey’s respondents, it is not surprising that a high proportion of them strongly recommend it to aspiring data scientists. 14,241 out of 17,821 people who answered this question made this recommendation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*uoXds-dgJRnBdvkWiOA5jg.png" /><figcaption>Above: Author’s image</figcaption></figure><p><strong>4. The past five years appears to have seen a big boost in machine learning practitioners</strong></p><p>Even though machine learning research has been ongoing since the <a href="https://ieeexplore.ieee.org/document/5392560?denied=">1950s</a>, a combination of factors has contributed to its recent popularity surge. For starters, computers are more affordable and powerful. Operations that once required specialized, expensive equipment can now be run from a personal laptop. Cloud computing infrastructure has become more advanced and democratised. Meanwhile, the sheer scale of big data delivered by the digital revolution has made it possible for businesses to ingest and analyse many more data points faster and cheaper than before.</p><p>Given the resurgence in machine learning, it is not surprising that just 12.67% of respondents said they do not use machine learning methods. The majority of respondents have done so within the past five years, including students.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*AP_VdNJuEPnakre_ilIv4w.png" /></figure><p>Many respondents with five or fewer years of coding experience use machine learning frameworks such as Scikit-learn and TensorFlow on a regular basis <em>(6746 / 10250 Scikit-learner users, and 4484 / 6934 TensorFlow users respectively)</em>. The former, which contains a robust library of tools in Python, can be used to perform classification, regression, and clustering among others. The latter, an open-source library released by Google in 2015, is used by companies like <a href="https://www.youtube.com/watch?v=tPb2u9kwh2w">AirBnB</a> to build and train models to solve problems like image classification.</p><p>Only 3% of respondents to this question said they do not use machine learning frameworks regularly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*Ysw86OfSfgWjAJ07AZXQXA.png" /><figcaption>Above: Author’s image</figcaption></figure><p><strong>5.</strong> <strong>It’s hard to tell if auto or partially auto machine learning tools remain a niche toolkit</strong></p><p>One challenging aspect of machine learning is determining the appropriate algorithm or tool to use, as there are <a href="https://ai.stanford.edu/~zayd/why-is-machine-learning-hard.html">trade-offs</a> with each approach. As you can imagine, this iterative process is resource-intensive and takes a long time. However, not many respondents said they used auto/partial auto machine learning tools such as <a href="https://machinelearningmastery.com/auto-sklearn-for-automated-machine-learning-in-python/">Auto SK Learn</a>, <a href="https://cloud.google.com/automl">Google Cloud AutoML</a>, and <a href="https://colab.research.google.com/github/phossen/autokeras/blob/master/AutoKeras.ipynb">Auto-Keras</a> on a regular basis, even though these can reduce the time associated with model development and accelerate the creation of production-ready models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*CCcpkmEYlIWfjZjgRkfv1A.png" /><figcaption>Above: Author’s image</figcaption></figure><p>It is worth noting that by this point in the survey <em>(Question 34 asked about automated or partially automated ML tools)</em>, the proportion of non-responses was above 95%. A sense of survey fatigue may have set in among many respondents by this point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*aTbW8O0piISC-XEar_py9A.png" /><figcaption>Above: Author’s image</figcaption></figure><p>Thanks for reading.</p><p><em>Data source: </em><a href="https://www.kaggle.com/c/kaggle-survey-2020"><em>Kaggle</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63311fbe4345" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Little-Known Trick in Hierarchical Clustering: Weights]]></title>
            <link>https://medium.com/data-science/a-little-known-trick-in-hierarchical-clustering-weights-762156a2fce0?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/762156a2fce0</guid>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[marketing-analytics]]></category>
            <category><![CDATA[hierarchical-clustering]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Mon, 04 Oct 2021 14:32:48 GMT</pubDate>
            <atom:updated>2021-10-04T14:32:48.416Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Recalibrate your dataset to answer a business question</em></h4><p>By Huey Fern Tay</p><p>With Greg Page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*E2JKGn6RBdOGx6ooXyaC-A.png" /><figcaption><em>Above: Photo by </em><a href="https://unsplash.com/@markkoenig?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Mark König</em></a><em> on </em><a href="https://unsplash.com/s/photos/lobster-land?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></figcaption></figure><p>When data is partitioned into meaningful groups, marketing analysts can see patterns in the way consumers behave and set the stage for the company to create targeted promotional campaigns. But clustering applications extend beyond marketing scenarios. For instance, scientists who study gene sequencing can use clustering to analyse evolutionary relationships among sequences in a group; property analysts can use this technique to understand how certain types of real estate should be priced relative to similar ones in the area.</p><p>In this article, I will demonstrate how a popular segmentation method, hierarchical clustering, can be used to create customer personas. Apart from that I will illustrate how weights can be used to recalibrate the dataset to reflect your current business priorities.</p><p>My dataset? The fictional, lovable Maine amusement park: Lobster Land.</p><p>Picture this scenario: With vaccination rates rising in many parts of the United States, Lobster Land wants to win customers back after putting the business on hold due to the COVID-19 pandemic. How can the park’s management target their customers?</p><p><strong>Step 1: Lose the categorical variables</strong></p><p>The first step is to drop the categorical variables <em>‘householdID’</em> and <em>‘homestate’.</em> <em>HouseholdID</em> is just a unique identifier, arbitrarily assigned to each household in the dataset. Since <em>‘homestate’</em> is categorical, it will not be suitable for use in this model, which will be based on Euclidean distance.</p><p>The drop() function from pandas will help us to accomplish this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c01a2a55f6016c93b9409abe503abb3/href">https://medium.com/media/6c01a2a55f6016c93b9409abe503abb3/href</a></iframe><p><strong>Step 2: Normalise the data</strong></p><p>Our dataset contains variables that are measured in different units, and on completely different scales. For instance, we have annual household income that ranges from $16000 to $213000, while the average email open rate per household ranges from 11% to 34%.</p><p>If we do not place this data on a ‘level playing field’, our clustering model would be dwarfed by the household income simply because those numbers are larger. We need to transform the data in such a way that makes the variables directly comparable with one another. Changing these numbers to z-scores, each with mean of 0 and standard deviation of 1, will do the trick.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd99590ec314b367388e8fd89ebe9a06/href">https://medium.com/media/dd99590ec314b367388e8fd89ebe9a06/href</a></iframe><p><strong>Step 3: Plot the dendrogram, decide the number of clusters, and create the clusters</strong></p><p>The dendrogram, which places the records on one axis and distances on the other, enables an analyst to see the way a nested clustering model forms. Dendrograms can be helpful for someone determining how many clusters to make. Keep in mind, however, that the chart is just a decision-making tool — there is no ‘right’ answer to the “how many clusters?” question, as this is an unsupervised learning process.</p><p>After some iteration and experimentation, I decided to cut the dendogram at the y=17 mark to generate a model with five distinct clusters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de6dfc037f452f435b5879015dd4d816/href">https://medium.com/media/de6dfc037f452f435b5879015dd4d816/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*QWcuzWKhzq_UIoSdkktp9w.png" /></figure><p>The code cell below contains instructions for setting up an agglomerative (bottom-up) clustering model with a distance threshold of 17.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5da91acbd4458b8ae0e832abbf9a85e8/href">https://medium.com/media/5da91acbd4458b8ae0e832abbf9a85e8/href</a></iframe><p>Notice the result above is a series of numbers ranging from 0 to 4 with square brackets on each end. Those numbers indicate the cluster assignment for each of our observations. We will need to convert these cluster assignments into a data frame, so that we can attach the numbers to the last column of our original dataset. This will make it easier for us to compare the clusters later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22362510b5e206dfdc417006b4a1c1f0/href">https://medium.com/media/22362510b5e206dfdc417006b4a1c1f0/href</a></iframe><p><strong>Step 4: Apply weights to each variable</strong></p><p>Up to this point, every variable in the clustering model is of equal importance — however, that does not reflect our business priorities. To make an educated guess about which groups of customers can be more easily convinced to return, we need to place more importance on certain factors such as the number of times they visited the park during the 2019 season.</p><p>With this objective in mind, this is how I re-weighted the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*np4nIyBKFQQdtmBxaVB0yw.png" /></figure><p>This re-weighting is a straightforward process. It is performed by simply multiplying each variable’s z-score values by a constant. A modeler using this process can easily adjust these weights in order to experiment with different outcomes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3c47629c5d5bc8fa589efb2b4c06fff/href">https://medium.com/media/d3c47629c5d5bc8fa589efb2b4c06fff/href</a></iframe><p><strong>Step 5: Re-cluster the data</strong></p><p>Applying different weights to variables changes the way some households are grouped. Placing a heavy importance on ‘<em>visits_2019’</em> has concentrated most of our frequent visitor households into just two particular clusters (Cluster 3 and Cluster 4). But wait…there’s more. This is just a glimpse of the changes that have happened.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdad5dd2ec583ae648c4158a8f332e9c/href">https://medium.com/media/fdad5dd2ec583ae648c4158a8f332e9c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*ijNUGR4a-rV2GmkmbbggWQ.png" /></figure><p><strong>Step 6: Examine the differences caused by the weighted data</strong></p><p>When a multiplier of 50x was applied to the number of visits, it causes the most frequent visitors (and least frequent visitors) to be more clearly separated from the rest of the pack. The large multiplier magnifies the differences in visits.</p><p>The tables below demonstrate this difference.</p><p>In the ward_cluster1 model (unweighted), the range in average visits across the clusters is just slightly greater than 7. We can see two frequent-visitor clusters with means above 16, while the other clusters hover around an average of 10. The standard deviation of these mean values is 3.33.</p><p>With the weighted clustering model, by contrast, the range of cluster averages becomes significantly bigger — for the ward_cluster2 model, the range of intra-cluster means jumps to more than 12. The standard deviation of mean visits per cluster jumps by nearly 44 percent, as the weighted model brings us higher highs, and lower lows, for our variable of greatest interest.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01e8666d56a1a0a40c5179d66a297f1a/href">https://medium.com/media/01e8666d56a1a0a40c5179d66a297f1a/href</a></iframe><p><strong>Step 7: Visualise the differences between the unweighted and weighted data</strong></p><p>Before we can colour code the charts based on the clusters, we need to convert them to categorical data.</p><p>Notice how the weighted data in the second density plot does not show much overlap between groups. In this second plot, the ‘super fans’ in the purple group (Cluster 4) are clearly separated from most of the dataset. The same applies to the pink group (Cluster 3) which represents the lower end of the ‘fan index’.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6644c669c9afe39a6e9e5cc4bfd1776/href">https://medium.com/media/d6644c669c9afe39a6e9e5cc4bfd1776/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*_agddyvJ_IHNrO4Uyf2vLg.png" /><figcaption><em>Above: A distribution plot of ‘visits_2019’ in the unweighted dataset. Notice some clusters are clumped together.</em></figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34ccca59ba04fe2d97a6d20a3f3cca83/href">https://medium.com/media/34ccca59ba04fe2d97a6d20a3f3cca83/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*ZNllmSMiWoVS4WVUP7qfhg.png" /><figcaption><em>Above: A distribution plot of ‘visits_2019’ in the weighted dataset. Notice there is less overlap between clusters and that the super fans in the purple cluster are much more distinct. The group in the red cluster stand out as well.</em></figcaption></figure><p><strong>Step 8: Examine the weighted clusters to create customer personas</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a762267f595518df0ebfe22950bcb1f/href">https://medium.com/media/0a762267f595518df0ebfe22950bcb1f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*PT2lEYoQbSzeSEFkyxw5Kw.png" /></figure><p>Up until this point, our demonstration has focused on <em>‘visits_2019’</em> which has the heaviest weight of 50%. While those results are telling on their own, we need to consider the effect the weights have on the remaining variables as well to get a clearer picture of Lobster Land’s clientele.</p><p>When we look at the average of all variables, we see visitor profiles that will help our co-workers in the marketing department create targeted campaigns. Here is the low-down on the groups that this model has identified among Lobster Land’s visitors:</p><p>· <strong>Die-hard fans: </strong>These families are the darlings of Lobster Land. Even though<strong> </strong>the average household lives 85.73 miles (around 137.8 km) away from Lobster Land, they do not mind the ‘hike’ down to the park. In 2019, these super families visited the park 18.03 times on average. They were the biggest spenders on online merchandise and were the most likely to open our marketing emails.</p><p>· <strong>Cashed up fans:</strong> The richest in the group. They too are strong supporters of Lobster Land, with an impressive average of nearly 13 visits throughout the 2019 season.</p><p>· <strong>Shopaholic loyalists:</strong> These customers should be commended for their dedication to Lobster Land because even though they live the furthest from the park, they averaged more than 15 visits on the season. This group of households has the second highest annual income on average. They are one of two clusters known for high online merchandise spending, and they have the second best e-mail open rate among the groups.</p><p>· <strong>Moderate fans: </strong>This group has the second-lowest visit average among the five clusters.</p><p>· <strong>The distant admirers:</strong> This group of families are probably the hardest to convince to come back. Even though they are likely to open marketing emails, they only visited the park 5.69 times in 2019. These households are relatively restrained in their spending as well.</p><p><strong>What does this mean for Lobster Land:</strong></p><p>Lobster Land management should pat themselves on the back for cultivating a strong bond with families. Prior to the COVID-19 pandemic, the average household visited the park every month even though the park is at least 84.47 miles (around 135.94 km) away — not exactly around the corner from where most people live.</p><p>Now that vaccination rates have risen, the good news is that it should not be very difficult to persuade people to revisit Lobster Land as people clearly have a deep sense of affection for the park.</p><p>The <em>‘die-hard fans’</em> will be the easiest to nudge and should be the first group of households the marketing department targets. These families are the most likely to open marketing emails, have a strong purchasing power, and clearly enjoy the Lobster Land experience therefore they are the easiest to push down the marketing funnel for any purpose: day trips to the park, the latest Larry the Lobster soft toys, staycations, birthday parties — you name it.</p><p>The <em>‘cashed-up fans’ </em>are also ardent fans who will not take much convincing to come back. They could be enticed to spend big at Lobster Land through activities like staycations because their estimated annual household income is the highest on average.</p><p>As previously pointed out, the <em>‘distant admirers’</em> will be the hardest to convince.</p><p><strong>Conclusion:</strong></p><p>A theme park in a place like Maine only has three months to make money every year before it becomes too cold. Given the short window of opportunity, it is imperative for these businesses to have hyper-focused marketing to ensure they maximise revenue for the year. At the same time, theme parks need to ensure they remain relevant to their audience by creating new rides and experiences.</p><p>The rebalancing example illustrated above is just the first step in getting people to return to the park after the COVID-19 disruption. To keep their finger on the pulse of the consumer, Lobster Land management needs to assess market conditions and consumer sentiment on a regular basis. They could conduct focus group discussions with their ‘super fans’ to test ideas for new attractions and analyse their overall marketing mix to determine whether their dollars are getting the most bang for their buck. In addition, A/B tests could also be applied to email newsletters to see what type of content works better as a ‘hook’.</p><p>Either way, tracking, testing, and iterating are must haves to sustain the future of Lobster Land for future generations.</p><p><strong><em>Data created by:</em></strong></p><p><em>Prof. Gregory Page, Senior Lecturer, Boston University</em></p><p><strong><em>Data source:</em></strong></p><p><em>Page, Gregory. (2021). Lobster Land Previous Passholders [comeback.csv]. Retrieved from </em><a href="https://drive.google.com/file/d/13bEScSEG95XluoSfbJ-H191oZZbgf49h/view?usp=sharing"><em>https://drive.google.com/file/d/13bEScSEG95XluoSfbJ-H191oZZbgf49h/view?usp=sharing</em></a></p><p><em>License: CC0</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=762156a2fce0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-little-known-trick-in-hierarchical-clustering-weights-762156a2fce0">A Little-Known Trick in Hierarchical Clustering: Weights</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When is it ok to impute missing values with a zero?]]></title>
            <link>https://medium.com/data-science/when-is-it-ok-to-impute-missing-values-with-a-zero-6d94b3bf1352?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d94b3bf1352</guid>
            <category><![CDATA[imputation]]></category>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-imputation]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Wed, 11 Aug 2021 16:06:58 GMT</pubDate>
            <atom:updated>2021-08-12T03:09:14.143Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Substituting missing data with a zero can sometimes be justified</em></h4><p><strong>By Huey Fern Tay</strong></p><p><strong>With Greg Page</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*qECz6hZyM5w9MXzrMzDzSA.png" /><figcaption>Image by: <a href="https://pixabay.com/photos/survey-opinion-research-voting-fill-1594962/">Andreas Breitling from Pixabay</a></figcaption></figure><p>Missing data appears for all sorts of reasons.</p><p>It happens when surveys are either too long or ask questions that are very personal (e.g. annual income, monthly expenses).</p><p>It occurs when machines break down.</p><p>Sometimes, it is simply the result of inconsistent record keeping.</p><p>Whatever the cause, one should think very carefully before filling in the blanks. Models built with the imputed data may deliver results that are strongly impacted by the newly altered values.</p><p>In a <a href="https://towardsdatascience.com/how-much-money-did-we-make-again-using-simputation-to-balance-the-books-f8e5c7b74f51">previous example</a> involving a fictional amusement park in Maine called Lobster Land, I showed that it was reasonable to estimate the revenue of an indoor gaming centre by looking at the number of day passes sold, because both factors had a strong and linear relationship.</p><p>But what about the other variable <em>‘precipitation’</em> which also contained missing information? Inference would not be applicable in this instance because rainfall does not depend on factors in our dataset such as the days of the week or sales revenue at SnackShack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*p5HRlY7yyuRp4FNBSKQGlQ.png" /><figcaption>Image by author</figcaption></figure><p>One reasonable way to treat the missing data would be to check the data to see if it rained most of the time. After all, if Maine was usually dry, it would not be right for us to assume rain occurred on the day our record keeping failed us.</p><p>To perform this verification using R, we use the <em>mfv() function</em> which stands for ‘most frequent value’.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5cefee0c0a47ad949d7d1037eef7c27/href">https://medium.com/media/e5cefee0c0a47ad949d7d1037eef7c27/href</a></iframe><p>Since rain was infrequent in Maine that year, we could address the missing values by inserting zeros into those cells.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/622a9b968828051013fb1d46d35e4235/href">https://medium.com/media/622a9b968828051013fb1d46d35e4235/href</a></iframe><p>Other times, simply using zero as an automatic substitute for missing values can be inappropriate. For instance, suppose we were missing values for coal consumed in the United States for the months shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*j-rHuQmt3NdBGRu-XKLGVQ.png" /><figcaption>Image by author</figcaption></figure><p>Firstly, imputing with zero would imply that coal mysteriously disappeared from the country’s energy mix for four months! While the NAs in the dataset are certainly frustrating, we would create a far larger problem here by suggesting that such consumption simply did not happen.</p><p>Secondly, this erroneous substitution would weaken multi-regression models. Note the difference in model strength when we switch from imputation with 0 to imputation with the mean value among the known months — the adjusted R-squared is immediately lifted from .5135 to .8586.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*OvhyCKBkLHLILCfEexG0qA.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*4gw9xKPUBRzVHo_8vsrpRQ.png" /><figcaption>Image by author</figcaption></figure><p>While not demonstrated here, a weak multi-regression model can have a negative chain effect on forecasting accuracy. Inaccurate planning made with those estimates can have a negative ‘ripple effect’ all the way down the supply chain. Once again, we see why it is important to proceed with caution when faced with missing data.</p><p>At the end of the day, there is no ‘one-size-fits-all’ solution for NA imputation. Replacement with zeros does have its merits, but as with the other approaches, context matters. Ultimately, the modeler needs to know the dataset, understand the problem, and be able to make the best decision for a particular set of circumstances.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d94b3bf1352" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-is-it-ok-to-impute-missing-values-with-a-zero-6d94b3bf1352">When is it ok to impute missing values with a zero?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using R’s simputation to handle missing values]]></title>
            <link>https://medium.com/data-science/how-much-money-did-we-make-again-using-simputation-to-balance-the-books-f8e5c7b74f51?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8e5c7b74f51</guid>
            <category><![CDATA[missing-values]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[imputation]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Sun, 25 Jul 2021 04:26:25 GMT</pubDate>
            <atom:updated>2021-11-11T01:47:32.719Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Holes in our dataset can be solved by making intelligent inferences from a linear model</em></h4><p>By Huey Fern Tay</p><p>With Greg Page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ioYy5VGkAB7krtHxYNv5GA.jpeg" /><figcaption><strong>Above: An amusement park; </strong><a href="https://pixabay.com/photos/amusement-carousel-amusement-park-2456905/"><strong>image by Jim Wellington from Pixabay</strong></a></figcaption></figure><p>To infer is to arrive at a conclusion based on evidence and reasoning.</p><p>In literature, this is done through contextual clues. Figures of speech such as “…you have a February face, so full of frost, of storm and cloudiness…” tell us that a person is gloomy because February is still a cold time of year in the northern hemisphere.</p><p>Similarly, in data science, intelligent guesses can be made based on our domain knowledge, linear modelling, or just common sense.</p><p>To illustrate this, I will use the example of a fictional amusement park in Maine called Lobster Land.</p><p>Suppose that you are working at Lobster Land as the resident data analyst. One day, the boss asks you to “fix the problem” after discovering some entries missing in a spreadsheet. With only a data dump in hand, and without knowing the context, where would you start?</p><p><strong>Step 1: Determine the severity of the missing data problem.</strong></p><p>We begin by installing the <em>visdat()</em> package in R. Applying the<em> vis_miss()</em> function to the raw data enables you to visualise the holes in your dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf7b4619ce71d90623b6f9b4cd991e92/href">https://medium.com/media/cf7b4619ce71d90623b6f9b4cd991e92/href</a></iframe><p>The chart below shows holes within two categories:<em> precipitation</em> and <em>GoldZoneRev</em>. The <em>precipitation</em> variable indicates the rainfall for a particular day, measured in inches. The <em>GoldZoneRev </em>variable, measured in dollars, indicates the revenue generated by Lobster Land’s indoor gaming area, where people win gold tickets that they can use to exchange for gifts.</p><p>Luckily, the gaps within <em>GoldZoneRev</em> only affect 5.66% of that variable’s data. In addition, the chart below tells us 3.77% of the data within the <em>‘precipitation’</em> column is missing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*yUnCiqyrDvdTubDSiNlGgw.jpeg" /></figure><p><strong>Step 2: Determine which factor(s) influence the revenue at the Gold Zone</strong></p><p>As I have written in a previous article, <a href="https://towardsdatascience.com/when-a-crazy-wind-wipes-away-our-records-what-do-you-do-a9f2aab3dc17">Last Observation Carried Forward</a> (LOCF) is one method we can use to treat missing values. It can work well in cases for which a previous observation is likely to offer an important clue about a missing value (as with daily weather data in a place with a seasonally changing climate). However, in this case, LOCF is not well-suited for the challenge at hand, as the revenue of an amusement park attraction can be influenced by several factors. The day of the week might matter, as might the overall volume of park visitors on a particular day. There is no reason for us to assume that we can make a reasonable imputation for this variable by simply using the previous day’s total.</p><p>A correlation matrix will tell us whether there is a relationship between other variables in the dataset, as well as the strength of those relationships.</p><p>To create the matrix, we use the <em>cor()</em> function. At the same time, we tell R to only use data that are classified as ‘numeric’ while ignoring the missing values. These are necessary instructions to prevent the command from failing. Finally, we round the results to two decimal places.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1916855a511047187ca890ee88adcc4/href">https://medium.com/media/c1916855a511047187ca890ee88adcc4/href</a></iframe><p>In the matrix shown below, the sizes of the circles correspond to the strength of the correlations. Darker shades of blue indicate positive correlations, whereas darker shades of red indicate negative ones. In this case, the number of day passes sold, the revenue from Snack Shack, and the number of hours Lobster Land staff work are three of the strongest factors that influence the day’s takings at the Gold Zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*4y_0Lkqbc9doh31y_nALWw.jpeg" /></figure><p><strong>Step 3: Estimate the day’s takings at the Gold Zone based on its relationship with one or more variables</strong></p><p>For the sake of simplicity in this example, we will try to estimate the revenue at the Gold Zone for those missing days based on one factor — the number of day passes sold.</p><p>The correlation matrix tells us the more day passes are sold at Lobster Land, the more money we make at the Gold Zone. In other words, there is a strong, positive, and linear relationship between those two factors (see chart below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*4DaLg_dFMd6crh2Cs5mONA.jpeg" /></figure><p>Since our missing values for <em>GoldZoneRev</em> lie within the range of our existing data, we can infer the day’s takings by looking at how many day passes were sold on the day. To do this, we use the <em>impute_lm()</em> function within R’s <em>simputation</em> package.</p><p>To make an intelligent guess about the day’s takings based on the number of day passes sold, we can run the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66fdc7b7e67b4808670281c1ef3e9f13/href">https://medium.com/media/66fdc7b7e67b4808670281c1ef3e9f13/href</a></iframe><p>In the code above, we first tell R to look within the ‘rawdata’ dataset. Then we add a column called <em>is_missing</em> to keep track of the cells that were originally blank. Next, we call the <em>impute_lm()</em> function. Within that function, we express<em> GoldZoneRev</em> as a function of <em>DayPass</em>.</p><p><strong>Step 4: Verify the results</strong></p><p>Day 13 is the first instance where the total daily revenue for Gold Zone was missing.</p><p>Our linear model tells us that the day’s revenue was $30914 on a day when 4151 day passes were sold. That estimate makes sense because it is only slightly lower than our takings on Day 7 when 4412 day passes were sold, and $33163 was generated at the Gold Zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*0-v7FZKULXmZNJGaGqbPjg.png" /></figure><p>So that leaves us with two questions. Could we have predicted Gold Zone’s missing revenue based on more than one factor? Yes, but this would have violated the parsimony principle — because the linear relationship between <em>DayPass</em> and <em>GoldZoneRev</em> was so strong, there was no need to introduce additional complications by adding other input variables.</p><p>Lastly, could we have used a similar linear model to predict missing values in the ‘<em>precipitation’ </em>column? The answer is no. Rain does not depend on the day of the week, the number of lobster rolls sold, or any other variables present in our dataset, so there would not be any suitable use of <em>impute_lm()</em> to fill those holes in our dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8e5c7b74f51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-much-money-did-we-make-again-using-simputation-to-balance-the-books-f8e5c7b74f51">Using R’s simputation to handle missing values</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Substituting missing data with the group average — why it’s good to be cautious]]></title>
            <link>https://medium.com/data-science/substituting-missing-data-with-the-group-average-why-its-good-to-be-cautious-d64bead7a029?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d64bead7a029</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[imputation]]></category>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[mean-imputation]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Fri, 09 Jul 2021 13:22:06 GMT</pubDate>
            <atom:updated>2021-07-10T05:15:12.701Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Substituting Missing Data with the Group Average: Why It’s Good to Be Cautious</strong></h3><h4>A categorical average may be better than LOCF or imputing with the overall mean, but this method has weaknesses that can affect your conclusions.</h4><p>By Huey Fern Tay</p><p>With Greg Page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*A52Ca7q6ekhUOCS5M0zaCA.png" /><figcaption>Photo by <a href="https://unsplash.com/@jojog1208?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Josephine Gasser</a> on <a href="https://unsplash.com/s/photos/tennis?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Tennis great Andre Agassi is used to the heat. He grew up in Las Vegas, where the summers get as hot as 39 to 41 degrees Celsius (102 to 105 degrees Fahrenheit) on average.</p><p>So when a man who is acclimatised to a subtropical desert climate describes the wilting summer in Melbourne, Australia as a “giant kiln”, that means something.</p><p>During the 2020 Australian Open tournament, there were some punishing days. On 31 January for example, the mercury hit 42.9C (109 degrees Fahrenheit). But you would not have known this if that value was missing from the Bureau of Meteorology’s historical records and there was no other way of finding out. If data was missing from 7 February and 72 other random days from that year, you would be faced with more questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*n4j83vfrKkcbUwxdvVSw4w.png" /></figure><p>Granted this is a highly unlikely scenario but there are several approaches you can use to make the best possible guess for those missing values.</p><p><a href="https://hftdata.medium.com/when-a-crazy-wind-wipes-away-our-records-what-do-you-do-a9f2aab3dc17"><strong>In a previous article</strong></a>, I described two other imputation methods: the Last Observation Carried Forward (LOCF) method, and replacement with an overall dataset mean. Here, I will demonstrate another approach: imputing missing values with a specific group average.</p><p>Using a specific group average for imputation is a more refined approach, compared with using the overall dataset average. Just as a detective would incorporate the available clues when attempting to solve a mystery, a data analyst can use group averages to make more reasonable estimations of likely values for replacing NAs.</p><p>Suppose, for example, we have university student records which include both ‘height’ and ‘gender’ among their attributes. Suppose also that some of the height values are missing, whereas all the gender values are known. If we impute the missing height values by the per-gender category average (175 cm for men, and 163 cm for women), we are much likelier to approximate the ‘true’ values than would be the case with a ‘one-size-fits-all’ mean imputation of the overall dataset mean.</p><p>Of course, the use of category means is far from perfect. Just as the height imputation method described above would miss outliers, using this approach for daily high temperatures cannot predict chart-busting days like 31 January 2020. But by replacing NAs with the monthly mean — especially in a place with a seasonally varied climate — our substituted value is far likelier to reflect reality, compared with an annual average.</p><p>To accomplish this, I used the dplyr package from R’s tidyverse to group the data by month. Next, I added the <em>impute_monthlymean</em> column to the dataset with the mutate() function. Then I gave R instructions on what to do within that function, as shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c84fa34b246e77e04ed499835fd86708/href">https://medium.com/media/c84fa34b246e77e04ed499835fd86708/href</a></iframe><p>This strategy is more accurate than imputing with the overall mean. Consider this: across the entire dataset, the monthly mean imputation values differed from the observed values by 0.59. This is an improvement of 27.6% (.813-.589)/.813), compared to the blunt approach of replacing all missing data with the mean temperature of the entire year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*o0iHY01K3j0rSg7kwlsUnA.png" /></figure><p>While the category-based imputation worked best among the three methods shown above, it will not always be ideal. In some situations, for example, researchers aim to establish correlations among variables. Simply imputing a missing value with the mean of that category will alter the correlation score and as a result, the conclusion about the relationship between variables.</p><p>In addition, mean imputation can distort the distribution of values within that variable. Even when performed with a category average, it will impact the variable’s standard deviation, and the inferences that can be drawn from mathematical models like ordinary-least squares regression. Once again, the insights gained will be biased, especially if the original dataset contained a large proportion of NAs. No matter what, a researcher who uses any form of imputation for NAs should clearly state this so that all readers can understand what was done, why it was done, and how it was done.</p><p>Many researchers in the healthcare industry avoid mean imputation due to the negative “domino effect” that it can have on inferential statistics (Kang, 2013). In other words, a decision to replace NAs with mean values at one step in the process can impact conclusions in subsequent steps.</p><p>So while missing values cause headaches by reducing the statistical power of the dataset, much thought needs to be given towards the treatment of missing data before diving into analysis and modeling. Defaulting to an easy solution such as mean imputation can present various problems later-on depending on the purpose of your analysis and the nature of your data.</p><p><strong>Reference:</strong></p><p>Kang, Hyun (24 May 2013). <em>‘The prevention and handling of missing data’</em>. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3668100/">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3668100/</a> . Korean Journal of Anaesthesiology.</p><p>Data source: <a href="http://www.bom.gov.au/climate/data/">Australian Bureau of Meteorology</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d64bead7a029" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/substituting-missing-data-with-the-group-average-why-its-good-to-be-cautious-d64bead7a029">Substituting missing data with the group average — why it’s good to be cautious</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When a crazy wind wipes away our records — what do you do?]]></title>
            <link>https://medium.com/data-science/when-a-crazy-wind-wipes-away-our-records-what-do-you-do-a9f2aab3dc17?source=rss-137b395910dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9f2aab3dc17</guid>
            <category><![CDATA[missing-values-in-r]]></category>
            <category><![CDATA[mean-imputation]]></category>
            <category><![CDATA[locf]]></category>
            <dc:creator><![CDATA[Huey Fern Tay]]></dc:creator>
            <pubDate>Sat, 03 Jul 2021 06:10:14 GMT</pubDate>
            <atom:updated>2021-07-11T19:27:47.018Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>When a crazy wind wipes away our records — what do you do?</strong></h3><h4>Should you assume that today’s temperature will be the same as yesterday’s or assume that the year’s average is a good benchmark?</h4><p>By Huey Fern Tay</p><p>with Greg Page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQWb8ONI17-HYUDIOWn7SQ.jpeg" /><figcaption><em>(Above: Rafael Nadal at the Australian Open; </em><a href="https://pixabay.com/images/id-288554/"><em>I</em>mage by moerschy</a><em>; created: 18 January 2012)</em></figcaption></figure><p>I love January in Melbourne, Australia.</p><p>January is when the tennis gods descend Down Under for the first tennis Grand Slam of the year — the Australian Open. While the tournament is on, Melbourne CBD is cloaked in a party atmosphere. The path along the Yarra River leading to Rod Laver Arena is filled with people sitting on benches and bean bags watching stars like Roger Federer or Serena Williams play on the big screen. Rod Laver Arena itself roars to life with every victory.</p><p>Those of you who are perhaps tempted to make the hike to this south-eastern city to see the action for yourself might wonder how hot Melbourne’s summer can get. There’s a fair chance that you might check Google, Weather.com or even Melbourne’s Bureau of Meteorology to get an idea of what to expect.</p><p>But what if, just what if, a crazy gust of wind swept away one-fifth of the values from the previous year’s records? Or what if the weather station at Melbourne Olympic Park broke down 20% of the time, causing those missing values? How could you plan ahead?</p><p>You might think that either of those scenarios is highly unlikely — and frankly, you’re probably right about that — but just in case, we decided to simulate such an event. After starting with a complete year’s worth of daily max temperatures for Melbourne, we randomly selected 20 percent of them for removal, leaving us with 73 NA values in that column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*TgN05sx3_YusBdd7eGZbzw.png" /></figure><p>Now, suppose we could not easily re-obtain the original values, and we needed to impute numbers for the max_temp variable so that our rows would once again be “complete cases” (complete cases are a requirement of certain modeling algorithms and visual techniques).</p><p>In such a scenario, one way to address that problem would be to use a statistical approach called Last Observation Carried Forward (LOCF), which means filling in the blanks with the most recent known value. For something seasonal, such as daily temperatures in Melbourne, this is a reasonable approach, especially when there isn’t an overwhelmingly large proportion of NAs. While Melbourne’s weather changes considerably across the span of an entire year, the day-to-day movements are tiny.</p><p>LOCF can be implemented with the <em>zoo</em> package in R, via the na.locf() function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0a303fd82e5bcaf980260309c3cb081/href">https://medium.com/media/c0a303fd82e5bcaf980260309c3cb081/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*YTG0PLZjh67PSO5Z-zfzwQ.png" /></figure><p>LOCF replaces each of our NAs with the most recent recorded observation. For cases in which there are two consecutive NAs — as we saw before with 29 January and 30 January, both are replaced with the observed value from the 28th.</p><p>An alternative to LOCF would be imputation with a single, central measure, such as the mean of the known observations across the entire dataset. Here, this would be accomplished by assigning the average 2020 temperature (i.e. 19.6 degrees Celsius, or 67.28 degrees Fahrenheit) to each empty record. We can do this with the impute_mean() function from the <em>naniar</em> package.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9e494c0bc26295106ff054f77774146/href">https://medium.com/media/c9e494c0bc26295106ff054f77774146/href</a></iframe><p>Now, we will compare the results from the LOCF method and the impute method against the original dataset.</p><p>Across the entire dataset, the LOCF values differed from the observed values by a mean absolute value of 3.99, with a median difference of 3, whereas the mean imputation values differed from the true observed values by a mean absolute value of 4.62, with a median of 3.8.</p><p>While both methods are straightforward to implement, easy to describe to an audience, and reliable with relatively predictable datasets, they also have some drawbacks. Imputation with a central measure is particularly troublesome with a widely-varied dataset. LOCF would not be appropriate for data that oscillates from period to period, such as the share prices of volatile stocks.</p><p>The drawbacks of both approaches would have been less obvious if we were dealing with a dataset with a narrower range of values such as temperature records from a tropical country, the length of various rose petals or the heights of several Chihuahuas.</p><p>That said, the accuracy of the impute method could have been improved if we had used the mean temperature of January to account for the gaps in that month.</p><p>But that’s a story for another time.</p><p><em>Data source: </em><a href="http://www.bom.gov.au/climate/data/"><em>Australian Bureau of Meteorology</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9f2aab3dc17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-a-crazy-wind-wipes-away-our-records-what-do-you-do-a9f2aab3dc17">When a crazy wind wipes away our records — what do you do?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>