<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Luce on Medium]]></title>
        <description><![CDATA[Stories by Matt Luce on Medium]]></description>
        <link>https://medium.com/@matt_24492?source=rss-6bee441a5de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RswuqkffEBuRdqi8YbxhOw.jpeg</url>
            <title>Stories by Matt Luce on Medium</title>
            <link>https://medium.com/@matt_24492?source=rss-6bee441a5de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:09:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matt_24492/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Analysis of Consumer Grocery Shopping Behaviors]]></title>
            <link>https://medium.com/@matt_24492/an-analysis-of-consumer-grocery-shopping-behaviors-846d034489e7?source=rss-6bee441a5de------2</link>
            <guid isPermaLink="false">https://medium.com/p/846d034489e7</guid>
            <category><![CDATA[fact-checking]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[business-data]]></category>
            <category><![CDATA[predictions]]></category>
            <dc:creator><![CDATA[Matt Luce]]></dc:creator>
            <pubDate>Thu, 09 Dec 2021 17:38:53 GMT</pubDate>
            <atom:updated>2021-12-09T20:30:41.431Z</atom:updated>
            <content:encoded><![CDATA[<h4>Fact-checking, predicting, and clustering using a Kaggle dataset.</h4><p><em>This study was completed as the final project for MIS 382N: Business Data Science at the University of Texas at Austin.</em></p><p><em>Group members: Matthew Luce, Adriana Van Tho, Teagan Webb, Sophia Falco, Pablo Pejla, Jin Huh, Jooyeong Kang, Narek Gevorgyan, and Manasvini Karthikeyan.</em></p><h3><strong>Project Description</strong></h3><p>We began this project with the desire to extract as many insights as possible from this Kaggle dataset. Thus, our problem statement is as follows: “How may we utilize a Kaggle dataset based on grocery store consumer shopping behaviors and tendencies to best cluster data points, predict values, and fact-check existing studies?” This dataset was collected to help businesses better understand customers and make better customer-minded decisions. Information in this dataset included basic demographics, family size, and income. Additionally, it included the amount spent on wines, fruits, meat products, fish products, sweets, and gold products in the last two years. Finally, it had information regarding purchasing mediums (IE, catalogs, online, in-store), and if a customer interacted with a promotion or sale. As many of the values in this dataset were continuous, we did some data manipulation and feature engineering to allow for greater opportunities. This included adding columns that categorically represented their education levels, income quartiles, family type, generation, and how recently they interacted with the store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57ADcVxlNbiVSrswuxOo9Q.png" /></figure><p>For informational purposes, the utilized Kaggle dataset had 29 columns and 2240 complete rows of data. Additional details about the dataset are posted below and linked in the sources section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2pVxmm2LnAH4n7XnCsR4A.png" /></figure><h3><strong>Clustering</strong></h3><p>To explore the consumer dataset we integrated the importance score metric, as described above, to determine features we would cluster with. Our data focus has revolved around the amount spent on wine by consumers, so our first step was to calculate the importance scores of all of the features dependent on the amount spent on wine. We found that the amount spent on gold and the amount spent on meat was the most correlated with the amount spent on wine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHtVmbxuafgz0VIkbbvNYQ.png" /><figcaption>Importance Features</figcaption></figure><p>To determine how many clusters would need to be created, we used the elbow graph method. This method plots the explained variation as a function of the number of clusters, then, at the elbow of the curve, shows the number of clusters to use (adapted from Wikipedia.com). Based on the elbow of the curve, we would utilize 3 clusters for this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jBfzJTkJR_Xv-7FlXCm2uA.png" /></figure><p>We performed a cluster for the amount spent on wine and meat but found the clusters to be inconclusive because there was no clear separation between the clusters. Similarly, the results for the amount spent on wine and gold products were inconclusive. This showed us that even though the variables, amount spent on meat and gold products, were the highest importance score in relation to the amount spent on wine, they were still not good clustering features for the dataset because they did not give us any insight. Even after feature engineering, we remained unable to cluster using these features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJalljDvRqgtVmn7F7J5Nw.png" /><figcaption>Amount spent on Wines vs Amount spent on Meat Products Clustering</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0z9N3JY_VPyW2RE-3XhfGQ.png" /><figcaption>Amount spent on Wines vs Amount spent on Gold Products Clustering</figcaption></figure><h3><strong>Fact-Checking</strong></h3><p>In the fact-checking section of our study, we hoped to find whether or not the information represented within this dataset aligned with the studies conducted by other research projects. Using research and statistics found within other studies, we then applied the same tests to our dataset to see if the claim was correct, relative to our dataset.</p><h4>Amount Spent on Wine and Children</h4><p>A study conducted by 4Children found that 57% of parents drink the same or more after their child is born. To confirm this claim in our dataset, we grouped customers by whether or not they have kids or teens and then calculated the average amount of wine consumption. Unlike the claim, in our dataset, customers who have no kids and teens are more likely to spend on wines than those who have kids or teens. For more detailed research, we conducted a statistical analysis to examine whether the means of wine consumption for these two groups are significantly different.</p><p>For the analysis, we chose to use the ANOVA test to compare the means of the amounts of wine consumption between the different groups. The null hypothesis of the ANOVA test is that the means of the groups are equal. To claim that the amount of wine consumption is significantly different between the groups, we need to reject this null hypothesis with the p-value of the test less than 0.05.</p><p>Before getting into the ANOVA test, we had to test for the underlying assumptions. There are two tests, one of which is the normality test to look at whether or not the data come from a normal distribution. The other is the homoscedasticity test to look at whether or not the variances of the groups are equal. If these assumptions are not violated through these tests, then we can use the ANOVA test. However, if these assumptions are violated, we need to use non-parametric tests like the Kruskal-Wallis test alternative to ANOVA.</p><p>First, for the normality test, we used the Shapiro-Wilk test which is one of the common normality tests. According to the results of the test, the two groups, one who have kids or teens and the other who do not, showed that these groups did not come from a normal distribution with a p-value that is close to 0.0.</p><p>As the two groups didn’t meet the normal distribution, we didn’t need to do the homoscedasticity test and we concluded that the assumption was violated and the ANOVA test was inappropriate. We used the Kruskal-Wallis test as an alternative to the ANOVA. According to the result of this test, the means of wine consumption amounts between the two groups were significantly different based on the p-value which was less than 0.05.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*HNT7LRuOEqJuo9dD7EFAvg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*5uV7rSaVs54MnoylTf7pDg.png" /><figcaption>Top: Shapiro-Wilk Test; Bottom: Kruskal-Wallis Test</figcaption></figure><h4>Amount Spent on Wine and Education</h4><p>Studies find that Americans with graduate degrees spend 10 times more per year on alcohol than those who did not attend college. According to the Bureau of Labor Statistics, those with graduate degrees spend approximately $992 per year on alcohol. This could be because those with higher education report higher incomes, which explains why they have more money to spend on alcohol. Additionally, a Gallup survey revealed that those who do not pursue higher education are also less inclined to drink.</p><p>Data from 2019 from the Bureau of Labor Statistics shows average annual alcohol spending by education level:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*mdlTolv1xSH0lKDMmpSY4w.png" /><figcaption>Alcohol Spending Data from 2019 from the Bureau of Labor Statistics</figcaption></figure><p>To confirm this claim, we followed those steps previously used. To divide the dataset into two groups, we separated the customers as an upper class by the median income and as higher education by college graduation.</p><p>Likewise the claim, in our dataset, customers who have higher income and education levels are likely to spend on wines than those who are not.</p><p>We conducted the Shapiro-Wilk test for these groups. Again, the two groups didn’t come from a normal distribution. Therefore, we used the Kruskal-Wallis test to compare the means of the amount of wine consumption between the two groups, and then, we got a p-value less than 0.05, which means that the groups were significantly different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*jKrVaG9Dr9mEoaL5-5xcEw.png" /><figcaption>Shapiro-Wilk Test</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*WzK-7sf62bJykMk32Zy9fw.png" /><figcaption>Kruskal-Wallis Test</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*6uhsDmsja9F0eSfobyTGLA.png" /></figure><h4>Income and Children</h4><p>The third claim which we analyzed came from ifstudies.org. The claim is as follows: “high-income families (relative to their countries income distribution) tend to have fewer children than low-income families.” To test this claim, we utilized our correlation matrix to see whether there was a positive or negative correlation between income and whether the family had kids. This was one of the sharpest negative correlations of our dataset. The matrix showed a correlation of -0.44 between income and kids at home. Therefore, we were successfully able to prove that our dataset supports the claim that high-income families (relative to their country&#39;s income distribution) tend to have fewer children than low-income families.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDlYcpwZbFeTtJnjOytT8A.png" /><figcaption>Correlation Matrix</figcaption></figure><h3>Prediction</h3><h4>Summary</h4><p>As for the prediction section, we aimed to predict different instances of the data, on responsiveness to ads, level of wine consumption, and identification of demographics. The classification was utilized in order to predict which customer is more prone to click on the promotion, and regression was utilized to predict the key demographics and the amount spent on wine by the customers. Together, it can be deployed in future datasets to help manage resources by advertising to the right customer. In other words, this can be used as a base for a good recommendation model.</p><h4>Classification for Advertisement Responsiveness</h4><p>To predict whether a customer would accept an ad and purchase after being exposed to it, we utilized a binary classification with the XGBoost Classifier and a Bayesian Optimization. After converting continuous data to categorical data, we were able to identify the consumer responsiveness with an Area Under the Curve (AUC) of 0.903. The model effectively predicts whether a new promotion will be accepted based on consumers’ demographic and behavioral data. Also, we found that converting the feature ‘birth year’ to ‘generation’, that is, from continuous to categorical, increased the performance of the classification but not by much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YiYVelsSKeqrHRWLdtIZ9g.png" /><figcaption>Classification Prediction Results</figcaption></figure><p>We reached a similar value of AUC when using CatBoost Classification with the Bayesian Optimization. This would be useful in understanding which type of customers the business should continue to spend resources on advertising, and the rest of the customers can be approached in a different way.</p><h4>Regression Prediction for Amount Spent on Wines</h4><p>When attempting to predict how much money was spent on wines, we found that our model can predict 94.6% of the variation of expenditure through some of the consumer’s demographic data and some of the behaviors:</p><ul><li>Their income quartile, whether they had kids or teens living with them, and the number of days since their last purchase.</li><li>That is, solely on the basis of demographic data, days since last purchase, and reaction to other marketing campaigns.</li></ul><p>As an alternate scenario where we also have access to expenditure data, we include the consumer behavior toward the amount of money spent on purchasing meat products, fish, fruit, and gold products. Counting with the behavioral data expands the R-squared score from 94.6% to 99.7%.</p><p>So, we can see the analysis possibilities with or without behavioral data, in addition to demographics.</p><p>First, we approached the problem with four different models. Namely XGBoost, CatBoost, Stacking classifier (base estimators: lasso, ridge, random forest and XGBoost; meta estimator: XGBoost) and LightBGM.</p><p>While all other models gave an accuracy between 75%-84%, with right tuning XGBoost performed the best.</p><p>We decided to move forward with XGBoost to predict for both classification and regression.</p><p>We performed feature engineering, after calculating the importance of each feature on the prediction. We dropped predetermined columns to observe how their presence or absence affected the final prediction. It is then, that we noticed that categorical values contributed more to the success of accurate predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ERsbiKgee3d8N7S6INb7g.png" /><figcaption>Regression Prediction Results</figcaption></figure><h3>Reverse Prediction</h3><p>Conversely, we tried to predict consumers’ demographics based on previous purchases.</p><p>Given the value of demographics to predict the money spent on specific products, we attempted to predict some of the customer’s characteristics.</p><p>As customers could be unwilling to share their personal data when they do not perceive any potential benefit, businesses can resort to statistical analysis to try to identify the persona behind their clients.</p><p>We trained some XGBoost Regression models on non-demographic consumer data to predict some demographic data such as the year of birth, income, and the numbers of their kids and teens.</p><p>Unfortunately, we failed to get a good model to predict the birth year of the customers. However, we succeeded in designing 3 models that can estimate the income, and number of kids and teens of a customer, pretty reliably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*OeafO23Xdk20u-CpQUitBQ.png" /><figcaption>Reverse prediction results</figcaption></figure><p>We were able to explain 91.5% of the variance in the customers’ income using our model after tuning the hyperparameters through a Bayesian optimization algorithm.</p><p>This tool can help businesses target their customers for tailored promotions while reducing the cost of having to acquire data.</p><h3>Sources</h3><p><a href="https://www.delish.com/food/news/a39367/new-parents-drink-more/">https://www.delish.com/food/news/a39367/new-parents-drink-more/</a></p><p><a href="https://www.techrepublic.com/article/data-privacy-is-a-growing-concern-for-more-consumers/">Data privacy is a growing concern for more consumers</a></p><p><a href="https://www.foodandwine.com/news/alcohol-spending-higher-education-study">https://www.foodandwine.com/news/alcohol-spending-higher-education-study</a></p><p><a href="https://www.forbes.com/sites/martyswant/2019/08/15/people-are-becoming-more-reluctant-to-share-personal-data-survey-reveals/?sh=32a678d1ed15">People Are Becoming More Reluctant To Share Personal Data, Survey Reveals</a></p><p><a href="https://ifstudies.org/blog/how-income-affects-fertility">https://ifstudies.org/blog/how-income-affects-fertility</a></p><p><a href="https://www.kaggle.com/imakash3011/customer-personality-analysis">Customer Personality Analysis</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=846d034489e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>