<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vivekananda Das on Medium]]></title>
        <description><![CDATA[Stories by Vivekananda Das on Medium]]></description>
        <link>https://medium.com/@vivdas?source=rss-964a6be9969------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TBfX9l7-vcP42HU2ZwVoHw.jpeg</url>
            <title>Stories by Vivekananda Das on Medium</title>
            <link>https://medium.com/@vivdas?source=rss-964a6be9969------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:49:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vivdas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[R Programming with AI: Generate R Code Using Easy Prompts]]></title>
            <link>https://medium.com/data-science-collective/r-programming-with-ai-generate-r-code-using-easy-prompts-68a92fb6fcea?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/68a92fb6fcea</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Wed, 26 Nov 2025 06:45:15 GMT</pubDate>
            <atom:updated>2025-12-01T07:39:03.065Z</atom:updated>
            <content:encoded><![CDATA[<h4>Practical Guide to Creating Data Products Quickly with AI</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xgf2Fdjm0lPYOrDA" /><figcaption>Photo by <a href="https://unsplash.com/@thisisengineering?utm_source=medium&amp;utm_medium=referral">ThisisEngineering</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This crash course teaches you how to use AI chatbots to generate R code.</p><p>I share with you prompts for creating the most common data products for survey data analysis. No installations or subscriptions are needed. <strong>A</strong>ll you need is a device that connects to the internet.</p><p>You can use these prompts by filling in the gaps (<em>____________) </em>based on your analysis goals and the dataset you are working with. Fundamentally, these prompts provide a structure that you can modify and expand to create the data products you need.</p><h3>Course Introduction</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fpkvv80XkRoY%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dpkvv80XkRoY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fpkvv80XkRoY%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8921fba7011179b7a576f198a2e92419/href">https://medium.com/media/8921fba7011179b7a576f198a2e92419/href</a></iframe><h3>Setting Up the Environment</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FCskVjBWXK2I%3Fstart%3D18%26feature%3Doembed%26start%3D18&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DCskVjBWXK2I&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FCskVjBWXK2I%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ab5676ec1d45f636e0b392db48d2f46/href">https://medium.com/media/1ab5676ec1d45f636e0b392db48d2f46/href</a></iframe><h3>Task 1: Simple Histogram</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The variable of interest is ____________, which means ____________. Only keep cases where the variable is ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a histogram of ____________ with ____________[Proportions/Frequencies] on the y-axis.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and an appropriate bin width. Do not add a title. Add a vertical line using contrasting color to show the median value. Add text showing the value of the median. Make sure you used accessibility principles.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FtzXLl_H4sBo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtzXLl_H4sBo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtzXLl_H4sBo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea301cb9cd418009119bbae1520423d8/href">https://medium.com/media/ea301cb9cd418009119bbae1520423d8/href</a></iframe><h3>Task 2: Grouped Histogram</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The variable of interest is ____________, which means ____________. Only keep cases where the variable is ____________.</em></blockquote><blockquote><em>The grouping variable is ____________, coded as ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a histogram of ____________ with ____________[Proportions/Frequencies] on the y-axis, displayed separately for each group using ____________ [Facets/Colors].</em></blockquote><blockquote><em>Convert the grouping variable into readable factor labels.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and an appropriate bin width. Do not add a title. Add a vertical line using contrasting color to show the median value. Add text showing the value of the median. Make sure you used accessibility principles.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FADaZdwJ4UOI%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DADaZdwJ4UOI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FADaZdwJ4UOI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/93d58470efb2e17bc65821db2203e008/href">https://medium.com/media/93d58470efb2e17bc65821db2203e008/href</a></iframe><h3>Task 3: Simple Boxplot</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The variable of interest is ____________, which means ____________. Only keep cases where the variable is ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a boxplot.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and an appropriate bin width. Do not add a title. Make sure you used accessibility principles.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FySpEL0g6MGA%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DySpEL0g6MGA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FySpEL0g6MGA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/73d0533e54d5ed93f5ca0ed4b0ee5539/href">https://medium.com/media/73d0533e54d5ed93f5ca0ed4b0ee5539/href</a></iframe><h3>Task 4: Grouped Boxplot</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The variable of interest is ____________, which means ____________. Only keep cases where the variable is ____________.</em></blockquote><blockquote><em>The grouping variable is ____________, which means ____________ and coded as ____________. Convert the grouping variable into readable factor labels.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a boxplot.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and an appropriate bin width. Do not add a title. Make sure you used accessibility principles.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FM2cCbMR4uAg%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DM2cCbMR4uAg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FM2cCbMR4uAg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/95ed0bed95318000f7a0d8ac487aee1a/href">https://medium.com/media/95ed0bed95318000f7a0d8ac487aee1a/href</a></iframe><h3>Task 5: Simple Scatterplot</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________, which means ____________ and the y-variable is ____________, which means ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a scatterplot of y-variable vs x-variable.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and high contrast colors following accessibility principles.</em></blockquote><blockquote><em>Do not add a title.</em></blockquote><blockquote><em>[Add a linear regression line with a contrasting color.]</em></blockquote><blockquote><em>[Add text showing the equation of the line and R-squared value on the plot.]</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FI_D7zcn9b6o%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DI_D7zcn9b6o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FI_D7zcn9b6o%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcf5c4b00ab53e6eef8764e3a9aa33b1/href">https://medium.com/media/bcf5c4b00ab53e6eef8764e3a9aa33b1/href</a></iframe><h3>Task 6: Grouped Scatterplot</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________, which means ____________ and the y-variable is ____________, which means ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>The grouping variable is ____________, coded as ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a scatterplot of y vs x, displayed separately for each group using ____________ [Facets/Colors].</em></blockquote><blockquote><em>Convert the grouping variable into readable factor labels.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and high contrast colors following accessibility principles.</em></blockquote><blockquote><em>Do not add a title.</em></blockquote><blockquote><em>[Add a linear regression line for each group with a contrasting color.]</em></blockquote><blockquote><em>[Add text showing the equation of the line and R-squared value for each group on the plot.]</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FglZsG73fBFA%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DglZsG73fBFA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FglZsG73fBFA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6857ddd96f9bb4bff99d2975b673c2bf/href">https://medium.com/media/6857ddd96f9bb4bff99d2975b673c2bf/href</a></iframe><h3>Task 7: Simple Line Chart</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________ , which means ____________, and coded as____________.</em></blockquote><blockquote><em>The y-variable is ____________ , which means ____________, and coded as ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a line chart showing the average of y-variable for each level of the x-variable. Show the dots as well. Dot size should be bigger than the width of line. Y axis value should begin from 0.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and high contrast colors following accessibility principles. Do not add a title. Add text labels showing the average value above each point. Make sure numbers are not cut off from screen.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5mzR9bCdUXg%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5mzR9bCdUXg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5mzR9bCdUXg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7ea927f504518153d493e5658705388/href">https://medium.com/media/c7ea927f504518153d493e5658705388/href</a></iframe><h3>Task 8: Grouped Line Chart</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________, which means ____________, and coded as ____________.</em></blockquote><blockquote><em>The y-variable is ____________, which means ____________, and coded as ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>The grouping variable is ____________, which means ____________, and coded as ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a grouped line chart showing the average of y-variable for each level of the x-variable, separately for each group. Use </em>____________<em>[Facets/Colors] to separate groups</em>.<em> to separate groups. Show the dots as well. Dot size should be bigger than the width of the line. Use contrasting colors for different groups.</em></blockquote><blockquote><em>Make sure the y-axis starts at 0 and has enough space above the highest point so text labels are not cut off.</em></blockquote><blockquote><em>Use large text size, clear axis labels, and high contrast colors following accessibility principles. Do not add a title. Add text labels showing the average value above each point.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxhlAdUraM4c%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxhlAdUraM4c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxhlAdUraM4c%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8e75b2bdae057dde1dff750cb25a2d1/href">https://medium.com/media/a8e75b2bdae057dde1dff750cb25a2d1/href</a></iframe><h3>Task 9: Simple Bar Chart</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________, which means____________, and coded as ____________.</em></blockquote><blockquote><em>The y-variable is ____________ , which means____________, and coded as ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a bar chart showing the ____________ [Counts/Averages/Percents] of ____________.</em></blockquote><blockquote><em>Convert the x-variable into readable factor labels.</em></blockquote><blockquote><em>Use large text size, high contrast colors, and clear axis labels following accessibility principles.</em></blockquote><blockquote><em>Do not add a title.</em></blockquote><blockquote><em>Add text labels on top of each bar showing the count, average, or percent, depending on the chart type.</em></blockquote><blockquote><em>Ensure bars are visually distinct and accessible.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEI_6_k4YKVA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEI_6_k4YKVA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEI_6_k4YKVA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8619a21047e1b60fdd410eb1ce7ae8e/href">https://medium.com/media/c8619a21047e1b60fdd410eb1ce7ae8e/href</a></iframe><h3>Task 10: Grouped Bar Chart</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The x-variable is ____________, which means____________, and coded as ____________.</em></blockquote><blockquote><em>The y-variable is ____________ , which means____________, and coded as ____________.</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>The grouping variable is ____________, which means ____________, and coded as ____________.</em></blockquote><blockquote><em>Show me R code using ggplot2 to create a bar chart showing the ____________ [Counts/Averages/Percents] of ____________. Separate groups by ____________ [Facets/Colors].</em></blockquote><blockquote><em>Convert the x-variable into readable factor labels.</em></blockquote><blockquote><em>Use large text size, high contrast colors, and clear axis labels following accessibility principles.</em></blockquote><blockquote><em>Do not add a title.</em></blockquote><blockquote><em>Add text labels on top of each bar showing the count, average, or percent, depending on the chart type.</em></blockquote><blockquote><em>Ensure bars are visually distinct and accessible.</em></blockquote><blockquote><em>Make sure I can export the image as a .png file with 300 dpi.</em></blockquote><blockquote><em>Do not assume I have any packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_LbtR4RJEWE%3Fstart%3D2%26feature%3Doembed%26start%3D2&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_LbtR4RJEWE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_LbtR4RJEWE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b522d39551dda3c8967d8ec69bed12d/href">https://medium.com/media/7b522d39551dda3c8967d8ec69bed12d/href</a></iframe><h3>Task 11: Summary Statistics Table</h3><h4>Prompt</h4><blockquote><em>The dataframe name is ____________.</em></blockquote><blockquote><em>The variables to summarize are ____________, which means ____________, and coded as ____________.</em></blockquote><blockquote><em>[The grouping variable is ____________, which means ____________, and coded as ____________.]</em></blockquote><blockquote><em>Filter out observations where ____________ is ____________.</em></blockquote><blockquote><em>Show me R code to:</em></blockquote><blockquote><em>Create a professional, publication-quality descriptive table, including<br> Means and standard deviations for continuous variables<br> Frequencies and percents for categorical variables<br> Clear variable labels and factor labels<br> Group comparisons if a grouping variable is provided</em></blockquote><blockquote><em>Ensure the table follows accessibility principles, including<br> Large readable font<br> Clear labels and headings<br> Appropriate formatting for print or screen</em></blockquote><blockquote><em>Export the output as an Excel (.xlsx) file, preserving the formatting as closely as possible.</em></blockquote><blockquote><em>Do not assume I have any packages installed. Install all required packages automatically.</em></blockquote><blockquote><em>Do not add a title to the table unless specified. Use base R functions to produce the table. Use an extra package only for exporting the excel file but not for table making. Do not assume I have these extra packages installed.</em></blockquote><h4>Video Showing an Example of Using the Above Prompt</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcvEtJgSqdxM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcvEtJgSqdxM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FcvEtJgSqdxM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fd201f67c3d453bc3340f898b63e53e/href">https://medium.com/media/0fd201f67c3d453bc3340f898b63e53e/href</a></iframe><h3>Extra Task #1: Conducting Weighted Analysis</h3><p>For a weighted analysis, first, use any of your prompts just as before.</p><p>Then, at the end, add this part:</p><blockquote><em>Use weights in the analysis. The weight variable is ____________ [type the name of the weight variable in your survey dataset].</em></blockquote><h4>Video Showing an Example of Mentioning the Use of Survey Weights in the Above Prompts</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWjrVOmxBJgM%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWjrVOmxBJgM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWjrVOmxBJgM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/66831a38f4fcf9709d2a9718a72e2981/href">https://medium.com/media/66831a38f4fcf9709d2a9718a72e2981/href</a></iframe><h3>Extra Task #2: Adding 95% Confidence Intervals in R</h3><p>For a weighted analysis, first, use any of your prompts just as before.</p><p>For adding 95% Confidence Intervals, first, use the prompts for bar charts or line charts just as before. Then, at the end, add this part:</p><blockquote><em>Show 95% Confidence Intervals.</em></blockquote><p>Also, you can remove the following line from your original prompt:</p><p>Add text labels showing the average value above each point.</p><h4>Video Showing an Example of Mentioning the Use of Survey Weights in the Above Prompts</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FgoNy46J1ot8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgoNy46J1ot8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FgoNy46J1ot8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/91357d1484c37aa2f594a076e19a4990/href">https://medium.com/media/91357d1484c37aa2f594a076e19a4990/href</a></iframe><h3>Extra Task #3: Verifying Code Written by AI</h3><h4>Video Showing an Example of How to Verify AI-Generated Code</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWNN5AwTq2NE%3Flist%3DPLLnN822fMufOWadkYkvz012nJ2R_TDQkC&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWNN5AwTq2NE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWNN5AwTq2NE%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ee84b513371d9d9b2b46099b3957332/href">https://medium.com/media/7ee84b513371d9d9b2b46099b3957332/href</a></iframe><p>Of course, we can do many more things with AI and R. But this is where we will end this crash course. I hope you can customize these prompts and create new prompts for your own needs.</p><p>Please let me know if you have any questions.</p><p>Thank you again for following along!</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68a92fb6fcea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/r-programming-with-ai-generate-r-code-using-easy-prompts-68a92fb6fcea">R Programming with AI: Generate R Code Using Easy Prompts</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Do Not Speak, People Do]]></title>
            <link>https://medium.com/data-science-collective/data-do-not-speak-people-do-0e7bb43223f0?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e7bb43223f0</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-literacy]]></category>
            <category><![CDATA[critical-thinking]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 03:35:22 GMT</pubDate>
            <atom:updated>2025-12-12T03:18:09.911Z</atom:updated>
            <content:encoded><![CDATA[<h4>Facts, Data, and the Narratives Between</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lt3mPeHOCF4s36xs6agYbA.png" /><figcaption><em>Image created using AI assistance via Microsoft Copilot</em></figcaption></figure><p>Few ideas have gained as much traction in recent decades as “data-driven decision-making,” now a near-sacred mantra across industries and governments. And that trust often pays off.</p><p>When a new virus spreads, health agencies rely on data to track infections and to guide where and how vaccines should be distributed. Businesses conduct experiments, such as testing whether a website button should be blue or red, to boost sales by fractions that add up to millions of dollars across countless transactions. Governments use data to forecast economic growth, allocate funding for schools, and prioritize infrastructure projects.</p><p>At its best, good data replace guesswork with evidence. It uncovers patterns we might not see otherwise.</p><h3>Leaving Whims and Biases Behind?</h3><p>Not so long ago, kings and emperors ruled with little or no data. They raised taxes because the treasury was getting empty or because a royal astrologer saw a bad omen in the sky. They declared wars on the advice of whispered promises that the gods would grant victory. There were no opinion polls to measure public sentiment. There was only raw power and the stories the powerful told to maintain it.</p><p>We like to think we have left that world behind us. We assume that decisions anchored in data are immune to the whims and biases that guided rulers in the past. Yet that belief conceals a risk of its own.</p><h3><strong>Not All Data Are Facts</strong></h3><p>Trouble begins when we forget that neither everything inside a dataset nor everything that comes out of it, such as graphs and tables, is a fact.</p><p>Let’s consider a simple measurement: height. Suppose Numberland, a fictional country, wants to recruit new soldiers. They record a recruit’s height as 168 centimeters. This is, in principle, a fact. Anyone with a tape measure should obtain (almost) the same result.</p><p>However, when an officer classifies the recruit as “short,” that label is not a fact in the same sense. Being “short” or “tall” only makes sense relative to a reference point. Short for what? For Numberland’s average adult male? For the average soldier? Compared to the neighboring country’s soldiers?</p><p>The measurement is largely objective; the assessment is mostly subjective. In datasets dealing with social and political topics, subjective assessments often appear in the same row as facts. When this occurs, it becomes easy to forget which data points are facts and which are subjective assessments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBweY6_vhMMnyHnieh7cBg.png" /></figure><p>Measurement of poverty is another example where this distinction becomes relevant. In Numberland, experts might define a single person as living in poverty if they earn less than 20,000 units of money a year. A dataset may show someone earning 19,999 units — that is a fact (assuming all their income has been accounted for). But calling that person “in poverty” is not a fact in the same sense, as it depends entirely on where the poverty line is drawn. Another single person earning 20,001 units is officially not “in poverty,” even if their situation is identical for all practical purposes.</p><p>The income is a fact; labeling that number as “living in poverty” is our choice, not nature’s. Recording poverty status in a dataset can be useful, but that does not make it an objective fact.</p><h4>The takeaway:</h4><blockquote>All facts can be expressed as data points, but not all data points are facts.</blockquote><h3><strong>Uncertainty in “Describing” the World</strong></h3><p>All measurements involve some uncertainty, and those targeting large, hard-to-track populations usually carry even more.</p><p>For example, a national survey might report that 53 percent of respondents support a new tax in Numberland. That figure appears precise, but survey data always contain uncertainty.</p><p>Suppose the survey randomly selected 500 people out of Numberland’s 5 million adult population. Because each random sample is a small snapshot of the population, results would vary from sample to sample. Another random sample might find 50 percent support, and a different one might find only 47 percent.</p><p>This variability is why survey results should include margins of error. Yet, in headlines, these nuances often vanish.</p><p>A reader who supports the tax sees “53 percent support the new tax” and assumes the people have spoken, because the number — implying majority approval — aligns with their expectation.</p><h4>The takeaway:</h4><blockquote>What appears to be a fact is often only an estimate, which by nature involves uncertainty.</blockquote><h3><strong>Uncertainty in “Explaining” the World</strong></h3><p>If describing the world with data is fraught with uncertainty, explaining why things happen is even trickier.</p><p>Suppose Numberland rolls out a Universal Basic Income (UBI) program. Every adult receives 2,000 units of money per month without any conditions attached.</p><p>Two years later, researchers find that average life satisfaction scores have risen from 7 to 8 on a 10-point scale. A government report announces: “The data prove UBI made people more satisfied.”</p><p>But did it? To determine whether the introduction of UBI caused this boost in life satisfaction, we must compare <em>what happened with UBI</em> against <em>what would have happened without UBI</em>.</p><p>And here is the catch: what does “without UBI” mean? Keep the old patchwork of welfare programs? Offer targeted aid only to those in need? Cut taxes by the same amount so people keep more of their paychecks? Spend the same money on childcare or health care instead?</p><p>Imagine two possible policy choices for Numberland. In the first, the government does nothing — no UBI and no new programs. In this scenario, life satisfaction stays at 7. With UBI, life satisfaction rises to 8, showing a positive effect of +1 compared to doing nothing. If instead the government invests the same amount in universal childcare, life satisfaction would rise to 9, an improvement of +2 over no program.</p><p>When we use “no program” as the baseline, UBI improves life satisfaction but less than universal childcare. If we take universal childcare as the baseline, the effect of switching to UBI is 8 minus 9, which equals −1. This means that, compared to universal childcare, UBI has a negative effect, lowering life satisfaction by 1 point.</p><p>The key takeaway is that the estimation of UBI’s impact depends entirely on which alternative scenario — the counterfactual — we choose to compare it against.</p><p>To estimate these counterfactual outcomes, social scientists often rely on randomized experiments or quasi-experiments. These methods help provide reasonably strong evidence about causal effects, but they are not perfect; all such estimates carry uncertainty and rest on assumptions that can be challenged.</p><ul><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-1d5f0e70a778">How to Explore the Effect of Doing Something? (Part 1)</a></li><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-part-2-228f857bb060">How to Explore the Effect of Doing Something? (Part 2)</a></li></ul><p>In social and political debates, however, the use of counterfactuals often becomes much more arbitrary. Supporters of a policy tend to pick comparisons that suggest positive effects, while opponents select those indicating negative or no effects. This strategic use of data shapes the narratives about whether a policy is seen as a success or a failure.</p><h4>The takeaway:</h4><blockquote>Real-world data alone do not settle debates about an event’s impact. We also need to consider counterfactual scenarios — the “what ifs” — which are estimates that carry uncertainty.</blockquote><h3>Data and the Stories We Tell</h3><p>In business, the purpose of using data is clear: to help companies maximize profit, a goal that (nearly) everyone involved agrees on. This alignment, together with the ability to track customer behavior accurately and run large-scale experiments easily, makes it possible to use data effectively. As a result, data-driven decisions often lead to better products, increased sales, and improved efficiency.</p><p>Social and political issues differ from business in two key ways. First, there is far less consensus among stakeholders, who often have competing goals. Second, unlike customer behavior on digital platforms, the behavior and opinion of the larger population cannot be tracked with the same accuracy, and large-scale experiments cannot be run that easily, both of which introduce greater uncertainty.</p><p>The combination of greater disagreement and uncertainty creates space for data to be collected and interpreted in divergent ways. Unsurprisingly, savvy leaders and experts use data not only to understand reality but also to craft narratives that support the perspectives they want to promote.</p><p><a href="https://medium.com/data-science-collective/propagandata-how-smart-people-strategically-use-data-to-create-narratives-67f8a531cc30">PropaganData: How Smart People Strategically Use Data to Create Narratives</a></p><p>For anyone following social and political debates, the key is to remember: not all data are pure facts. Often, what we encounter are narratives built around data — narratives designed to promote or attack particular viewpoints. Therefore, being aware of the relationship among facts, data, and narratives can help us become smarter consumers of data products and better judges of the policies shaping our society.</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><ul><li><a href="https://vivdas.medium.com/why-good-people-disagree-dissecting-social-media-conflicts-d646e4c3d01c">Why Good People Clash: Dissecting Social Media Conflicts</a></li><li><a href="https://vivdas.medium.com/does-money-buy-happiness-4ad92109c303">Does Money Buy Happiness?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e7bb43223f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/data-do-not-speak-people-do-0e7bb43223f0">Data Do Not Speak, People Do</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PropaganData: How Smart People Strategically Use Data to Create Narratives]]></title>
            <link>https://medium.com/data-science-collective/propagandata-how-smart-people-strategically-use-data-to-create-narratives-67f8a531cc30?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/67f8a531cc30</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-literacy]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-propaganda]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Sat, 21 Jun 2025 05:40:10 GMT</pubDate>
            <atom:updated>2025-11-15T07:36:17.011Z</atom:updated>
            <content:encoded><![CDATA[<h3>PropaganData: How Smart People Strategically Use Data to Shape Narratives</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rmxBzcXwakbIuVP7" /><figcaption>Photo by <a href="https://unsplash.com/@miguel_photo?utm_source=medium&amp;utm_medium=referral">Miguel Henriques</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a student of social sciences for the last five years, I have come across many models, theories, and frameworks from different disciplines that attempt to explain why people behave the way they do.</p><p>Among all of these, the one I find most useful is the utility maximization framework. This framework, traditionally used by economists, argues that rational agents behave in ways that maximize their utility — the satisfaction or value they derive from their decisions.</p><p>This framework is especially useful for understanding the ways in which many experts, implicitly or explicitly aligned with particular groups, use data to construct narratives that portray their groups more favorably than competing ones. Here, by experts, I mean individuals with subject-matter authority who often act as intermediaries between data and public opinion.</p><p>Why would experts engage in constructing narratives that favor their groups rather than uncovering the truth, regardless of whom it favors? Are they not meant to remain objective?</p><p>The reality is they are human first, experts second — and humans are social animals. Belonging to a group helped our ancestors survive in a hostile planet.</p><p>In less hostile conditions, where survival is not the main concern, we still know our lives are finite. According to <a href="https://psycnet.apa.org/record/1990-14626-001">Terror Management Theory</a>, awareness of mortality can evoke feelings of meaninglessness and despair, prompting us to seek connection with something larger, something that will outlast us. This may be where the need to belong — and to assert group superiority — becomes so powerful. Because this process often happens unconsciously, experts — even those well-versed in the scientific method — may not recognize their strategic use of data as a form of bias.</p><p>If you pay close attention, you will notice these strategies rely on exploiting the ambiguities and uncertainties inherent in human experience. Some critics might label them as <em>propaganda</em>, but I would rather embellish the term and call it <em>propagandata</em> — the strategic use of data to craft preferred narratives.</p><p>In this article, I describe three common strategies experts use to shape narratives. Of course, I use <em>hypothetical</em> examples to illustrate these strategies. I do not wish to jeopardize my life or livelihood. After all, this is how a rational agent, when faced with a survival threat, would seek to maximize their utility.</p><h3>Strategy 1: Choice of Quantitative versus Qualitative Evidence</h3><p>Let’s consider a fictional place where the government recently changed hands. There is growing concern that violent crimes are on the rise.</p><p><strong>Pro-government experts</strong> cite quantitative evidence (e.g., percentages and averages) to downplay the issue:</p><blockquote>“Yes, there have been some isolated incidents. But crime affects less than 0.01% of the population. The overwhelming majority are safe.”</blockquote><p><strong>Anti-government experts</strong>, on the other hand, go qualitative. They highlight individual stories to humanize the victims:</p><blockquote>“Meet the family of someone who lost their loved one last month. Hear what that loss has done to their lives.”</blockquote><p>While one side uses numbers to reassure, the other uses stories to evoke moral outrage. One side says, “Look at the data”; the other says, “Look at this family.”</p><p>By strategically switching between quantitative and qualitative forms of evidence, experts shape narratives that serve the interests of their groups.</p><h3>Strategy 2: Selection of Quantitative Measures</h3><p>Suppose the government in a hypothetical place raises the flat income tax rate from 10% to 11%.</p><p>Now imagine two politically active groups:</p><ol><li><strong>Pro-tax group: </strong>Those who favor expansive government involvement and support higher taxes</li><li><strong>Anti-tax group: </strong>Those who prefer limited government and oppose tax increases</li></ol><p>Among the general public, the common preference is predictable: get the most government benefits while paying the least in taxes.</p><p>So, how do the two expert camps frame the same tax hike?</p><p><strong>Pro-tax experts</strong> emphasize the modesty of the change:</p><blockquote>“It is <em>just a one-percentage-point</em> increase.”</blockquote><p>They want people to think the hike is <em>small and reasonable</em>.</p><p><strong>Anti-tax experts</strong>, on the other hand, use a different measure. Instead of percentage point change, they talk about <em>percentage change</em>. Their narrative might go like this:</p><blockquote><em>“If your taxable income was 10,000 units, you used to pay 10,000 × 0.10 = 1,000.</em></blockquote><blockquote><em>Now, at 11%, you pay 10,000 × 0.11 = 1,100.</em></blockquote><blockquote><em>That is 100 more units or a </em>hefty 10% increase<em> in taxes relative to what you paid before.”</em></blockquote><p>So, is it a <em>minor </em>1-point increase or a <em>burdensome </em>10% hike? Which group is correct?</p><p>Well, both. And that is what makes propagandata so elusive. You might argue that the anti-tax group has a stronger point because people will experience a real increase in tax burden. But you cannot say the pro-tax group is wrong for pointing out that the rate increased by only one percentage point.</p><p>People often focus more on the magnitude of a measure than its technical meaning. Experts know this and choose the measure that makes a change seem small or large, depending on what they want people to think.</p><h3>Strategy 3: Preference-Dependent Reference</h3><p>Suppose an international watchdog finds that, over the past year, for every 100 units of public money raised by the government, 50 units were lost to corruption.</p><p><strong>Anti-government experts</strong> seize the moment:</p><blockquote>“Half of our hard-earned money has been looted under this administration. These criminals caused this catastrophe!”</blockquote><p><strong>Pro-government experts</strong>, though seemingly cornered, respond with a clever counterpoint:</p><blockquote>“Wait! To say this government caused the issue assumes that, had another party been in power, there would be less corruption. That’s not reality.”</blockquote><p>They may cite historical data showing that under previous governments in the past 5, 10, 20, or even 50 years (whichever temporal choice helps get the magic reference number), on average, corruption was about 55 units per 100. Using the data, they argue:</p><blockquote>“Rather than blaming the government for the (50 units per 100) corruption, critics should congratulate us for reducing corruption (by 5 units per 100). We are making things better.”</blockquote><p>Once again, it is not just about the data. No one, not even the experts defending the government, denies the veracity of the 50 units of corruption per 100 units of public money raised.</p><p>What matters is what you compare it to. Changing reference points can turn bad news into a success story, or make a good outcome look terrible.</p><p>Some experts build their careers around tactfully selecting reference points to shape narratives. When attacking the other group, they might use an ideal benchmark — say, zero corruption or crime. When defending their own group, they shift to a different reference point, such as the average corruption or crime rate over a period of their choice. This tactic — where reference points are chosen based on what someone wants to argue — is what I call <em>preference-dependent reference</em>.</p><p>After years of observing how group loyalty shapes the strategic use of data, I have stopped asking who is telling the truth.</p><p>I now ask: who chose the measures and the reference points, and as rational agents trying to maximize their group-centric utility, what story were they trying to tell?</p><p>That is where the real story begins.</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><ul><li><a href="https://vivdas.medium.com/data-do-not-speak-people-do-0e7bb43223f0">Data Do Not Speak, People Do</a></li><li><a href="https://vivdas.medium.com/why-good-people-disagree-dissecting-social-media-conflicts-d646e4c3d01c">Why Good People Clash: Dissecting Social Media Conflicts</a></li><li><a href="https://vivdas.medium.com/the-secret-to-happiness-be7bcf0be017">The Secret to Happiness: Where Buddhism and Behavioral Economics Meet</a></li><li><a href="https://vivdas.medium.com/does-money-buy-happiness-4ad92109c303">Does Money Buy Happiness?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67f8a531cc30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/propagandata-how-smart-people-strategically-use-data-to-create-narratives-67f8a531cc30">PropaganData: How Smart People Strategically Use Data to Create Narratives</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Good People Clash: Dissecting Social Media Conflicts]]></title>
            <link>https://vivdas.medium.com/why-good-people-disagree-dissecting-social-media-conflicts-d646e4c3d01c?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/d646e4c3d01c</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[conflict]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[social-media]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Wed, 01 Jan 2025 05:57:26 GMT</pubDate>
            <atom:updated>2025-10-23T22:28:34.951Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last decade, social media has evolved from a space for connecting with friends and family into a battleground of ideas, where fierce debates and ideological clashes are the new norm.</p><p>Many assume that conflicts arise because others ignore <em>“the facts,” </em>believing that harmony would follow if everyone simply accepted the truth.</p><p>But is that always the case? I would argue otherwise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AOgvJwUFZhsL_f6S" /><figcaption>Photo by <a href="https://unsplash.com/@henniestander?utm_source=medium&amp;utm_medium=referral">Hennie Stander</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>The Three Pillars of Disagreement</strong></h3><p>During my graduate studies in sustainability, I noticed recurring patterns of disagreement among experts despite sharing a common goal of building a sustainable future and dealing with the same numbers.</p><p>They often struggled to agree on what sustainability meant. If they reached a consensus on the definition, they still disagreed on how to measure it. Even when there was agreement on both definition and measurement, conflicts arose over whether specific actions promoted or undermined sustainability.</p><p>Later, as I started my academic journey in social sciences, I realized this pattern was not unique to sustainability or only among academic researchers.</p><blockquote>Most human disagreements — academic or non-academic, online or offline — boil down to one or more of three core issues: morality, measurement, and causality.</blockquote><blockquote>Digging even deeper, all three issues share a common root — uncertainty.</blockquote><h3>Uncertainty in Morality: What Should We Do?</h3><p>Morality may seem solid at first glance — a clear guide to what is right and wrong. However, it is a social construct shaped by time and place. A behavior deemed moral in one era or society might be viewed as immoral in another. This fluidity means that their interpretations can diverge widely even when people align on basic moral principles.</p><p>For example, although people share the moral principle of valuing societal well-being, they often disagree on how to maximize it. Some emphasize immediate needs, arguing that alleviating poverty and improving the lives of those suffering today should be the priority. In contrast, others contend that we must prioritize tackling challenges like climate change to ensure long-term well-being for present and future generations.</p><p>These conflicts are definitely not about facts. In fact, they are not even about a lack of moral values but divergent interpretations of shared principles.</p><h3>Uncertainty in Measurement: How Should We Define and Quantify?</h3><p>Suppose we reached a consensus on interpretations of shared principles. Would the fighting stop? Not really!</p><p>The next hurdle is measurement: defining and quantifying key constructs.</p><p>Take fairness, for instance. How do we define it? Is it treating everyone equally or giving more to those most in need? And once defined, how do we measure it? By outcomes? By opportunities?</p><p>Even in everyday situations — like dividing household chores, splitting a bill, or deciding on work promotions — people often disagree on what’s “fair.”</p><p>The measurement uncertainties create situations where even morally aligned people find reasons to fight. Definitions and practical benchmarks for fairness, justice, or responsibility are neither universal nor absolute. These uncertainties fuel endless debates in personal and societal contexts.</p><h3>Uncertainty in Causality: What Causes What?</h3><p>Even with consensus on morality and measurement, we are left with the biggest challenge: causality.</p><blockquote>Causality is difficult, if not impossible, to prove because it requires <em>counterfactual</em> evidence: observing what would happen if we could simultaneously exist in two (or more) identical worlds, differing only in the factor of interest.</blockquote><p>Take Bee and Sea, a married couple wondering if their relationship makes them happier. To know for sure, they need to <em>compare their current happiness to a counterfactual world where everything else is the same as in this world, except they are not married</em>. Such comparisons are impossible because counterfactuals are, by definition, <em>unobservable</em>.</p><blockquote>I find that causality lies at the heart of most human disagreements, both online and offline. This is because people often intuitively construct counterfactuals to confirm their preconceived notions. This often leads to disagreements that <em>fundamentally revolve around whose counterfactual is more plausible</em>.</blockquote><h3>Fighting in the Age of Social Media</h3><p>Presumably, uncertainty around morality, measurement, and causality has fueled human disputes throughout history. But social media adds a dangerous twist to the issue.</p><p>Instead of helping people realize and appreciate uncertainty — a fundamental aspect of the human experience — it amplifies certainty. It makes individuals more convinced of their perceptions of morality, measurement, and causality and more hostile to others’ perceptions. People begin to see those who disagree with them as existential threats rather than collaborators in gaining a more useful understanding of the human experience.</p><p>To make things worse, in places that lack strong social and political institutions, the hostile environment — created by a toxic interaction between algorithms and tribalistic human nature — turns intellectual disagreements into violence.</p><h3>A Path Toward Civil and Productive Fighting</h3><p>What can we do to reduce the toxicity of human disagreements in the age of social media?</p><p>I want to focus on two things.</p><blockquote>First, experts need to help people realize that disagreements often arise from uncertainty about morality, measurement, and causality rather than bad intentions or not believing in facts. This is easier said than done; nevertheless, a necessity.</blockquote><blockquote>Second, experts should create accessible online content that helps people gain a fundamental understanding of statistical and causal inference — essential tools to deal with uncertainty.</blockquote><p>While knowledge of these concepts may not directly address moral dilemmas, a broader understanding of ideas like confidence intervals can illustrate that measurements and predictions — even those made by the brightest minds using the best resources — carry inherent uncertainty.</p><p>Also, learning the circumstances under which correlation implies causation can prevent oversimplified conclusions. Understanding counterfactual reasoning can improve the quality of debates on societal issues such as gun laws and immigration policy.</p><p>Of course, these efforts will not end hostile online and offline conflicts. This is, at least partially, because most humans naturally prefer certainty over uncertainty. Certainty provides comfort, reinforces collective identities, and makes life meaningful. Even many scientists, who are trained to embrace uncertainty, succumb to this tendency.</p><p>Nevertheless, if we can help people recognize and appreciate uncertainty and equip them with the appropriate tools, we may foster greater compassion and reduce the hostility in human disagreements.</p><p>That is not asking for too much, is it?</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><ul><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-1d5f0e70a778">How to Explore the Effect of Doing Something? (Part 1)</a></li><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-part-2-228f857bb060">How to Explore the Effect of Doing Something? (Part 2)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d646e4c3d01c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interpreting the Coefficients of a Regression with an Interaction Term (Part 2)]]></title>
            <link>https://vivdas.medium.com/interpreting-the-coefficients-of-a-regression-with-an-interaction-term-part-2-4c1178422aa7?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c1178422aa7</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[interaction]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 07:48:58 GMT</pubDate>
            <atom:updated>2025-10-23T22:28:53.306Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Detailed Explanation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cgOXfu-zMEj8q13X" /><figcaption>Photo by <a href="https://unsplash.com/@dengxiangs?utm_source=medium&amp;utm_medium=referral">Choong Deng Xiang</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article discusses how to interpret the coefficients of the predictors of a linear model that includes a two-way interaction between <strong>two continuous predictors </strong>or <strong>two binary predictors.</strong></p><p>This article is a continuation of the following article which discusses the interpretation of the coefficients of the predictors of a linear model that includes a two-way interaction between<strong> a continuous predictor and a binary predictor</strong>.</p><p><a href="https://vivdas.medium.com/interpreting-the-coefficients-of-a-regression-model-with-an-interaction-term-a-detailed-748a5e031724">Interpreting the Coefficients of a Regression Model with an Interaction Term: A Detailed…</a></p><h3>Interaction between two continuous predictors</h3><p>Let’s consider a hypothetical example in which we are predicting the <strong>Salary </strong>of employees in a particular profession based on their <strong>years of experience </strong>and <strong>years of education </strong>— two continuous predictors — and an interaction between them.</p><p>The model can be written as:</p><blockquote>Salary = b0 + b1 * Experience + b2 * Education + b3 * Experience * Education (1)</blockquote><p>For this example, I will generate some fake data and estimate the model in R:</p><pre># Set the seed <br>set.seed(123)<br><br># Generate data for Experience (in years) and Education (in years)<br>n &lt;- 10000  # Number of observations<br>Experience &lt;- round(runif(n, 0, 20), 1)  # Experience between 0 and 20 years<br>Education &lt;- sample(16:24, n, replace = TRUE)  # Education as an integer between 16 and 24 years<br><br># Define the coefficients<br>beta_0 &lt;- 30000  # Intercept<br>beta_1 &lt;- 2000   # Coefficient for Experience<br>beta_2 &lt;- 1500   # Coefficient for Education<br>beta_3 &lt;- 300    # Coefficient for Interaction<br><br># Generate Salary using the model<br>Salary &lt;- beta_0 + beta_1 * Experience + beta_2 * Education + beta_3 * Experience * Education<br><br># Add some random noise to make the data more realistic<br>Salary &lt;- Salary + rnorm(n, mean = 0, sd = 5000)<br><br># Combine into a dataframe<br>data &lt;- data.frame(Experience, Education, Salary)<br><br># Estimate the model<br>model &lt;- lm(Salary ~ Experience * Education, data = data)<br>summary(model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*9V6hV4kAsoNy9VDvW0q9Cw.png" /><figcaption>Results of the estimated model (screenshot from RStudio)</figcaption></figure><p>So, the estimated coefficients are:</p><p><strong>b0</strong> (intercept) = 30293.391</p><p><strong>b1</strong> (coefficient of <strong>Experience</strong>) = 2027.885</p><p><strong>b2</strong> (coefficient of <strong>Education</strong>) = 1482.503</p><p><strong>b3</strong> (coefficient of <strong>Experience * Education</strong>) = 298.573</p><h4><strong>Interpretation of b0:</strong></h4><p>The intercept represents the predicted (average) base <strong>Salary</strong> (which is <strong>$30293.391</strong>) when both <strong>Experience</strong> and <strong>Education</strong> are 0.</p><p>However, in this case, having 0 years of <strong>Education</strong> is not realistic because education levels in our dataset range from 16 to 24 years. Therefore, the intercept does not have a meaningful real-world interpretation by itself.</p><p>Instead, it is the starting point for calculating salary in the context of the regression equation. It essentially adjusts the <strong>Salary</strong> prediction based on the values of other variables in the model.</p><h4>Interpretation of b1:</h4><p>In this model, <strong>b1</strong> (the coefficient of <strong>Experience)</strong> shows how much (<strong>$2027.885</strong>) <strong>Salary</strong> is, on average, associated with each additional year of <strong>Experience</strong> when <strong>Education</strong> is 0.</p><p>This is because when <strong>Education</strong> is 0, the model in equation (1) essentially reduces to:</p><blockquote>Salary = b0 + b1 * Experience</blockquote><p>However, having zero <strong>Education</strong> is not realistic given that our <strong>Education </strong>variable ranges from 16 to 24 years. Therefore, direct interpretation of <strong>b1</strong> is not practically meaningful.</p><p><em>[Note: In a linear model without an interaction term, the coefficient of </em><strong><em>Experience</em></strong><em> would tell us the average increase in </em><strong><em>Salary</em></strong><em> for each additional year of </em><strong><em>Experience</em></strong><em>, assuming </em><strong><em>Education</em></strong><em> is held constant at any level, not necessarily zero.]</em></p><h4>Interpretation of b2:</h4><p>In this model, <strong>b2</strong> (the coefficient of <strong>Education</strong>) shows how much (<strong>$1482.503</strong>) <strong>Salary</strong> is, on average, associated with each additional year of <strong>Education</strong> when <strong>Experience</strong> is 0.</p><p>This is because the model essentially reduces to:</p><blockquote>Salary = b0 + b2 * Education</blockquote><p><em>[Note: In a linear model without an interaction term, the coefficient of </em><strong><em>Education</em></strong><em> would tell us the average increase in </em><strong><em>Salary</em></strong><em> for each additional year of </em><strong><em>Education</em></strong><em>, assuming </em><strong><em>Experience</em></strong><em> is held constant at any level, not necessarily zero.]</em></p><h4>Interpretation of b3:</h4><p>The interaction term indicates how the association between <strong>Experience</strong> and <strong>Salary</strong> depends on <strong>Education</strong>, and vice versa. Specifically, for every additional year of <strong>Education</strong>, the association between <strong>Experience</strong> and <strong>Salary</strong>, on average, increases by <strong>$298.573</strong>, and for every additional year of <strong>Experience</strong>, the association between <strong>Education</strong> and <strong>Salary</strong>, on average, increases by <strong>$298.573</strong>.</p><p>This means that the association between <strong>Experience</strong> and <strong>Salary</strong> is stronger when <strong>Education</strong> is higher, and similarly, the association between <strong>Education</strong> and <strong>Salary</strong> is stronger when <strong>Experience</strong> is higher.</p><h3>Interaction between two binary predictors</h3><p>Let’s consider another hypothetical example in which we are predicting the <strong>Salary </strong>of employees in a particular profession based on their <strong>Job Type (Part Time and Full Time) </strong>and <strong>Certification Status (No Certification and Certification) </strong>— two binary predictors — and an interaction between them.</p><p>Here is how the binary predictors are coded:</p><p><strong>Job Type</strong> == 1 if someone works <strong>Full Time</strong> and <strong>Job Type </strong>== 0 if someone works <strong>Part Time</strong></p><p><strong>Certifaction Status</strong> == 1 if someone has <strong>Certification </strong>and <strong>Certification Status </strong>== 0 if someone has <strong>No Certification</strong></p><p>The model we are estimated can be written as:</p><blockquote>Salary = b0 + b1 * Job Type + b2 * Certification + b3 * Job Type * Certification (2)</blockquote><p>Again, I will generate some fake data and estimate the model in R:</p><pre># Load necessary packages<br>library(dplyr)<br><br># Set seed <br>set.seed(123)<br><br># Generate data with 10,000 observations<br>n &lt;- 10000  # Number of observations<br><br>data &lt;- data.frame(<br>  Job_Type = sample(0:1, n, replace = TRUE),        # 0 = Part-time, 1 = Full-time<br>  Certification = sample(0:1, n, replace = TRUE)     # 0 = No Certification, 1 = Certification<br>)<br><br># Create a salary variable based on a hypothetical model<br>data &lt;- data %&gt;%<br>  mutate(<br>    Salary = 35000 + <br>             7000 * Job_Type + <br>             4500 * Certification + <br>             3000 * Job_Type * Certification + <br>             rnorm(n, mean = 0, sd = 5000)  # Adding some random noise<br>  )<br><br># Estimate the model<br>model &lt;- lm(Salary ~ Job_Type * Certification, data = data)<br><br># Show the model summary<br>summary(model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*i7U7es-HufDGZV4d7FZEhg.png" /><figcaption>Results of the estimated model (screenshot from RStudio)</figcaption></figure><p>So, the estimated coefficients are:</p><p><strong>b0</strong> (intercept) = 34831.15</p><p><strong>b1</strong> (coefficient of <strong>Job Type</strong>) = 7189.05</p><p><strong>b2</strong> (coefficient of <strong>Certification</strong>) = 4629.10</p><p><strong>b3</strong> (coefficient of <strong>Job Type</strong> * <strong>Certification</strong>) = 2859.34</p><h4>Interpretation of b0:</h4><p>This is the predicted (average) <strong>Salary</strong> for someone with a part-time job and no certification (both Job Type and Certification are 0).</p><h4>Interpretation of b1:</h4><p>In this model, <strong>b1</strong> (the coefficient of <strong>Job Type)</strong> shows the difference in predicted (average) salary (<strong>$7189.05</strong>) between those with <strong>Full Time</strong> Job (<strong>Job Type ==1)</strong> and <strong>Part Time</strong> Job (<strong>Job Type==0</strong>) when <strong>Certification</strong> is 0 (i.e., for those who have <strong>No Certification</strong>).</p><p>This is because when <strong>Certification</strong> is 0, the model in equation (2) essentially reduces to:</p><blockquote>Salary = b0 + b1 * Job Type</blockquote><h4>Interpretation of b2:</h4><p>In this model, <strong>b2</strong> (the coefficient of <strong>Certification)</strong> shows the difference in predicted (average) salary ($<strong>4629.10</strong>) between those with <strong>Certification</strong> (<strong>Certification ==1</strong> and <strong>No Certification</strong> (<strong>Certification==0</strong>) when <strong>Job Type </strong>is 0 (i.e., for those who have <strong>Part Time Job</strong>).</p><p>This is because when <strong>Job Type</strong> is 0 the model in equation (2) essentially reduces to:</p><blockquote>Salary = b0 + b2 * Certification</blockquote><h4>Interpretation of b3:</h4><p>In this model, <strong>b3</strong> estimates the following:</p><p>[Average<strong> Salary</strong> for those <strong>with Certification</strong> and <strong>Full Time Job</strong> —<strong> </strong>Average<strong> Salary</strong> for those <strong>with No</strong> <strong>Certification </strong>and<strong> Full Time Job]</strong> — [Average<strong> Salary</strong> for those <strong>with Certification</strong> and <strong>Part Time Job</strong> — Average<strong> Salary</strong> for those <strong>with</strong> <strong>No Certification </strong>and <strong>Part Time Job]</strong> = $2859.34</p><p>A simplified explanation of <strong>b3</strong> is that it indicates how the <strong>association between Job Type and Salary</strong> changes depending on <strong>Certification, </strong>and how the <strong>association between Certification and Salary</strong> changes depending on <strong>Job Type</strong>.</p><p>It is easier to understand the interpretation of <strong>b3</strong> mathematically.</p><p>Let’s think about four groups depending on the values of <strong>Job Type</strong> and <strong>Certification</strong> and estimate salary for each group by replacing the values of the two predictors in equation (2):</p><p><strong>Group #1: Part-time and No Certification</strong> (Job Type = 0, Certification = 0):</p><p>Salary = b0</p><p><strong>Group #2: Full-time and No Certification</strong> (Job Type = 1, Certification = 0):</p><p>Salary = b0 + b1</p><p><strong>Group #3: Part-time with Certification</strong> (Job Type = 0, Certification = 1):</p><p>Salary = b0 + b2</p><p><strong>Group #4: Full-time with Certification</strong> (Job Type = 1, Certification = 1):</p><p>Salary = b0 + b1 + b2 + b3</p><p><strong><em>Difference</em> </strong>in average<strong> Salary </strong>between<strong> </strong>those<strong> with and without certification </strong>for<strong> full-time jobs</strong>:</p><p>(b0 + b1 + b2 + b3) − (b0 + b1) = <strong>b2 ​+ b3​</strong></p><p><strong><em>Difference</em> </strong>in average<strong> Salary </strong>between those<strong> with and without certification </strong>for<strong> part-time jobs</strong>:</p><p>(b0 + b2) − b0 ​= <strong>b2​</strong></p><p>The coefficient of the interaction<strong> </strong>term <strong>(b3)</strong> is the<strong> <em>difference</em> </strong>between the<strong> <em>two differences</em>:</strong></p><p>(<strong>b2 + b3</strong>) − <strong>b2</strong> = <strong>b3</strong></p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><ul><li><a href="https://towardsdatascience.com/what-does-it-mean-to-control-for-something-in-multiple-regression-744880620988">What Does It Mean to Control for a Variable in Regression?</a></li><li><a href="https://towardsdatascience.com/interpreting-regression-coefficient-wish-i-had-known-this-before-177dabb27f58">Interpreting Regression Coefficient: Wish I Had Known This Before</a></li><li><a href="https://vivdas.medium.com/understanding-regression-output-in-r-a-lesson-for-absolute-beginners-part-1-e45f43343a4c">Understanding Regression Output in R: A Lesson for Absolute Beginners (Part 1)</a></li><li><a href="https://vivdas.medium.com/understanding-regression-output-in-r-a-lesson-for-absolute-beginners-part-2-b4291c908099">Understanding Regression Output in R: A Lesson for Absolute Beginners (Part 2)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c1178422aa7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Survey Data in R: A Crash Course (Part 3)]]></title>
            <link>https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-3-4048a5d98dd4?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/4048a5d98dd4</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[histograms]]></category>
            <category><![CDATA[box-plot]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Mon, 25 Dec 2023 07:18:34 GMT</pubDate>
            <atom:updated>2025-10-23T22:29:22.951Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step lesson</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pJs4b0GW3hppLUeE" /><figcaption>Photo by <a href="https://unsplash.com/@towfiqu999999?utm_source=medium&amp;utm_medium=referral">Towfiqu barbhuiya</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome to the third and final part of our crash course on survey data analysis in R!</p><blockquote>**You can find the video lectures of this crash course on the course website😊**</blockquote><p><a href="https://www.vivekanandadas.com/surveycourse.html">Survey Data Analysis in R</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9biGaJwR0ik%3Fstart%3D12%26feature%3Doembed%26start%3D12&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9biGaJwR0ik&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9biGaJwR0ik%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href">https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href</a></iframe><p>In this article, I share with you the following lessons:</p><blockquote><strong><em>Task 1:</em></strong><em> Merging multiple segments of a survey by common columns</em></blockquote><blockquote><strong><em>Task 2:</em></strong><em> Summarizing a continuous variable</em></blockquote><p>If you missed the first two parts of this course, here are the links to them:</p><ul><li><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-1-9dfa4b110115">Analyzing Survey Data in R: A Crash Course (Part 1)</a></li><li><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-2-03da4ec119ec">Analyzing Survey Data in R: A Crash Course (Part 2)</a></li></ul><p>For the two tasks, I use data from the <strong>American Time Use Survey (ATUS) 2022 </strong>conducted by the U.S. Bureau of Labor Statistics. ATUS datasets are publicly available here: <a href="https://www.bls.gov/tus/data.htm">https://www.bls.gov/tus/data.htm</a></p><p>The goal of our analysis is to get a sense of <strong>the duration of the walking trips made by Americans in 2022</strong>.</p><p><em>**To avoid confusion, note that the unit of analysis here is time spent walking “per trip” (not “per person per day”)**</em></p><h3>Task 1: <em>Merging multiple segments of a survey by common columns (e.g., Respondent IDs)</em></h3><p>In the <a href="https://www.bls.gov/tus/data/datafiles-2022.htm">webpage</a> that contains ATUS 2022 data, you can find the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIsSXvAi5LjRvn0zfTVqgA.png" /></figure><p>Note that there are multiple segments of the ATUS data. For example, ATUS 2022 Respondent file, ATUS 2022 Roster file, etc.</p><p>To merge multiple segments of ATUS into one dataframe, follow these steps:</p><p>1) Download the <a href="https://www.bls.gov/tus/datafiles/atusresp-2022.zip">ATUS 2022 Respondent file (zip)</a> and the <a href="https://www.bls.gov/tus/datafiles/atusact-2022.zip">ATUS 2022 Activity file (zip)</a>. These files are stored in zip folders. Click on them and copy the .DAT files located inside the zip folders.</p><p><em>[You can merge other segments of the ATUS as well. But, I will keep things simple and create a combined dataset using these two files.]</em></p><p>2. Create a folder in your local drive. Paste the two .DAT files in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRL-242eNtPLsjcP5ZRfFQ.png" /></figure><p>3. Next, create an R script and save it in the folder where all the ATUS files are located. I named my script <strong>atus_analysis_code.</strong></p><p>4. Next, click: <strong>Session → Set Working Directory → To Source File Location</strong></p><p>5. Once you do the above, you will see the following in your console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*5UNE35fkiDk3zpKXNXv-Bw.png" /></figure><p>Copy the<strong> setwd(…….)</strong> part and paste the code into your script.</p><p>Next time you run the code, it will direct R to the right place.</p><p>6. Load the <strong>dplyr</strong> and <strong>ggplot2</strong> packages.</p><pre>#Load packages<br>library(dplyr)<br>library(ggplot2)</pre><p>7. Import the two ATUS files by running the following code:</p><pre>#Importing the two segments of ATUS 2022<br>act &lt;- read.csv(&quot;atusact_2022.DAT&quot;)<br>resp &lt;- read.csv(&quot;atusresp_2022.DAT&quot;)</pre><p>8. Amazing! The two ATUS files have been imported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*GyqyQggDo9MPsc1jeGwOQA.png" /></figure><p>9. Merge the two dataframes using the <strong>merge ( ) </strong>function.</p><p>Note that the only common column(variable) between the two dataframes is <strong>TUCASEID. </strong>Therefore, we want to combine them based on this common column.</p><pre>#Merging the two segments of ATUS 2022 by the common column<br>data &lt;- merge(resp, act, by=&quot;TUCASEID&quot;)</pre><p>If you had multiple common columns — for example, X and Y other than <strong>TUCASEID</strong> — you would have used the <strong>c( )</strong> function in the code to include them all in the merging process. Below, I share the code applicable to that hypothetical scenario:</p><pre>data &lt;- merge(resp, act, by=c(&quot;TUCASEID&quot;,&quot;X&quot;,&quot;Y&quot;))</pre><p>10. Great! Now, I can see the merged dataframe, called <strong>data</strong>, in the global environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*aWJjmbeaOX9Gtg-GsF-5Eg.png" /></figure><h3>Task 2: <em>Summarizing a continuous variable</em></h3><p>To investigate our research question, we need three variables from the ATUS, which are <strong>TEWHERE, TUACTDUR</strong>, and <strong>TUDIARYDAY.</strong></p><p>Below, I present a detailed description of these variables by taking screenshots from the <a href="https://www.bls.gov/tus/dictionaries/atusintcodebk22.pdf">ATUS codebook</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YOWrjxbwXh0fqm7LiABn7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*euKfqtfn8geOVI6oMIfVDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZyS8s0aeyvSGdghwfg9Cqg.png" /></figure><h4>Create a Subset that Has Observations on Walking Trips</h4><p>First, let’s create a subset that contains observations on walking trips (i.e., TEWHERE==14):</p><pre>#Creating a subset of walking trips<br>walking &lt;- data %&gt;% filter(TEWHERE==14)</pre><h4>Five Number Summary</h4><p>We can use the <strong>fivenum( )</strong> function to get summary statistics (minimum, 1st Quartile, Median, 3rd Quartile, and Maximum) of the <strong>TUACTDUR </strong>variable, which shows the time respondents spent in walking trips (in minutes):</p><pre>#Five Number Summary of Time Spent Walking in a Day<br>fivenum(walking$TUACTDUR)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*3IC6ghXZml21PCiz8ZSu3A.png" /></figure><p>Based on the above, we learn the following:</p><p>Minimum=1; 1st Quartile=5; Median=8; 3rd Quartile=15; Maximum=362 (in minutes)</p><h4>Histogram</h4><p>Next, we create a histogram by using functions from the <strong>ggplot2</strong> package.</p><pre>#Histogram<br><br>ggplot(walking, aes(x=TUACTDUR))+<br>  geom_histogram(binwidth = 1)+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nBevw3TT7X_RIhUE5YySrg.png" /></figure><p>It looks like the tail of the histogram is really long, although there are hardly any observations above 100 minutes. Therefore, we can limit the x-axis values between 0 and 100 by using the <strong>xlim( )</strong> function:</p><pre>#Histogram<br><br>ggplot(walking, aes(x=TUACTDUR))+<br>  geom_histogram(binwidth = 1)+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;)+<br>  xlim(0,100)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HEWjM4JlZ2YgkCb-mri_Q.png" /></figure><p>We can play around with the <strong>binwidth ( )</strong> function. For example, below I present a histogram using <strong>binwidth</strong> = 5.</p><pre>#Histogram<br><br>ggplot(walking, aes(x=TUACTDUR))+<br>  geom_histogram(binwidth = 5)+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;)+<br>  xlim(0,100)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tNE-d8qjuB6_EG4SYKssVQ.png" /></figure><p>Depending on the context of your research, you should pick your binwidth. It is difficult to provide a generic response on what the ideal binwidth is.</p><h4>Boxplot</h4><p>Next, we can create a boxplot by using the following code:</p><p><em>[Note the similarity between the code for a boxplot and the code for a histogram]</em></p><pre>#Boxplot<br><br>ggplot(walking,aes(x=TUACTDUR))+<br>  geom_boxplot()+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;,fill=&quot;Diary Day&quot;)+<br>  xlim(0,100)+<br>  theme(axis.text.y=element_blank(), #eliminates y axis text<br>        axis.ticks.y=element_blank()) #eliminates y axis ticks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xxzEsZI40xvaL1Pq_hVhw.png" /></figure><p>The above <strong>boxplot</strong> gives us a sense of the distribution of the <strong>TUACTDUR </strong>variable for the whole sample (i.e., respondents in the <strong>walking</strong> dataframe).</p><p>Now, let’s pretend that we want to incorporate the <strong>TUDIARYDAY </strong>variable in the boxplot and get separate boxplots for each day. We can do that by using the following code:</p><pre>#Boxplot<br><br>ggplot(walking,aes(x=TUACTDUR,fill=factor(TUDIARYDAY)))+<br>  geom_boxplot()+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;,fill=&quot;Diary Day&quot;)+<br>  xlim(0,100)+<br>  theme(axis.text.y=element_blank(),<br>        axis.ticks.y=element_blank())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oN0FJ9A8niZa65b_rFweig.png" /></figure><p>Looks good!</p><p>Lastly, we can bring one more change to the above boxplot.</p><p>To make it more professional, we can change the levels of the <strong>TUDIARYDAY </strong>variable from numbers to the names of the days so that our readers understand the boxplot more easily.</p><p>To do that, we run the following code:</p><pre>#Boxplot<br><br>#Recoding the levels of the TUDIARYDAY variable<br>walking$TUDIARYDAY &lt;- recode(walking$TUDIARYDAY,&quot;1&quot;=&quot;Sunday&quot;, <br>                                                &quot;2&quot;=&quot;Monday&quot;,<br>                                                &quot;3&quot;=&quot;Tuesday&quot;,<br>                                                &quot;4&quot;=&quot;Wednesday&quot;,<br>                                                &quot;5&quot;=&quot;Thursday&quot;,<br>                                                &quot;6&quot;=&quot;Friday&quot;,<br>                                                &quot;7&quot;=&quot;Saturday&quot;) <br><br>#Reordering the levels of the TUDAIARYDAY variable and creating the boxplot<br>walking %&gt;% mutate(TUDIARYDAY = factor(TUDIARYDAY, levels = c(&quot;Sunday&quot;,<br>                                                              &quot;Monday&quot;,<br>                                                              &quot;Tuesday&quot;,<br>                                                              &quot;Wednesday&quot;,<br>                                                              &quot;Thursday&quot;,<br>                                                              &quot;Friday&quot;,<br>                                                              &quot;Saturday&quot;)))%&gt;%<br>  ggplot(aes(x=TUACTDUR,fill=factor(TUDIARYDAY)))+<br>  geom_boxplot()+<br>  theme_classic()+<br>  labs(x=&quot;Duration (in minutes)&quot;,fill=&quot;Diary Day&quot;)+<br>  xlim(0,100)+<br>  theme(axis.text.y=element_blank(),<br>        axis.ticks.y=element_blank())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SgGZKzIDyHusAZvP4vLfQA.png" /></figure><p>This is the end of our crash course.</p><p>Please let me know if you have any questions.</p><p>Thank you again for following along!</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4048a5d98dd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Survey Data in R: A Crash Course (Part 2)]]></title>
            <link>https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-2-03da4ec119ec?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/03da4ec119ec</guid>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analyzing-survey-data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[survey-data-analysis-in-r]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Sat, 21 Oct 2023 03:28:00 GMT</pubDate>
            <atom:updated>2025-10-23T22:29:52.756Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step lesson</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3s348JefKMUdOgIA" /><figcaption>Photo by <a href="https://unsplash.com/@jannerboy62?utm_source=medium&amp;utm_medium=referral">Nick Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome to the second part of our crash course on survey data analysis in R!</p><blockquote>**You can find the video lectures of this crash course on the course website😊**</blockquote><p><a href="https://www.vivekanandadas.com/surveycourse.html">Survey Data Analysis in R</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9biGaJwR0ik%3Fstart%3D12%26feature%3Doembed%26start%3D12&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9biGaJwR0ik&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9biGaJwR0ik%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href">https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href</a></iframe><p>In this article, I share with you the following lessons:</p><blockquote><strong><em>Task 1:</em></strong><em> Importing a .csv file from your local drive</em></blockquote><blockquote><strong><em>Task 2:</em></strong><em> Merging multiple rounds/phases of a survey by common columns</em></blockquote><blockquote><strong><em>Task 3:</em></strong><em> Creating a line chart</em></blockquote><blockquote><strong><em>Task 4:</em></strong><em> Estimating an empirical model using linear regression</em></blockquote><blockquote><strong><em>Task 5: </em></strong><em>Saving graphs and tables</em></blockquote><blockquote><strong>Extra Task: </strong><em>Creating a grouped line chart</em></blockquote><p>If you missed the first part of this course, here is the link to it:</p><p><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-1-9dfa4b110115">Analyzing Survey Data in R: A Crash Course (Part 1)</a></p><p>For these tasks, I use data from the <strong>Survey of Household Economics and Decisionmaking (SHED)</strong> <strong>2017–2022</strong> conducted by the Federal Reserve Board. These datasets are publicly available here: <a href="https://www.federalreserve.gov/consumerscommunities/shed_data.htm">https://www.federalreserve.gov/consumerscommunities/shed_data.htm</a></p><p>The goal of our analysis is to explore how <strong>people’s perceptions of the economic condition</strong> evolved between 2017 and 2022.</p><h3><strong><em>Task 1:</em></strong><em> Importing a .csv file from your local drive</em></h3><p>Let’s follow these steps:</p><p>1. Download the 6 SHED files (2017–22) from the Federal Reserve Board website. These files are stored in zip folders. Click on them and copy the .csv file located inside the folder.</p><p>2. Create a folder in your local drive. Paste all the .csv files in it. Here is how it looks on my computer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v8LRJ782b6j7C_GsREmXcw.png" /></figure><p>3. Next, create an R script and save it in the folder where all the SHED files are located. I named my script <strong>shed_code</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3AguQle2E9_SxUH1SzESgA.png" /></figure><p>4. Next, click: <strong>Session → Set Working Directory → To Source File Location</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p_8U5MVwpQHld0OCIbc5LA.png" /></figure><p>5. Once you do the above, you will see the following in your console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sl69sbWz63UEJw_y1gbb1Q.png" /></figure><p>Copy the<strong> setwd(…….)</strong> part and paste the code into your script.</p><p>Next time you run the code, it will direct R to the right place.</p><p>6. Import the 6 SHED files by running the following code:</p><pre>#Import the 6 SHED datasets and name them <br>shed2017&lt;-read.csv(&quot;public2017.csv&quot;)<br>shed2018&lt;-read.csv(&quot;public2018.csv&quot;)<br>shed2019&lt;-read.csv(&quot;public2019.csv&quot;)<br>shed2020&lt;-read.csv(&quot;public2020.csv&quot;)<br>shed2021&lt;-read.csv(&quot;public2021.csv&quot;)<br>shed2022&lt;-read.csv(&quot;public2022.csv&quot;)</pre><p>Great! The 6 SHED files have been imported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*a1kMMz1Y5mXzQLltZIjH1Q.png" /></figure><h3><strong><em>Task 2:</em></strong><em> Merging multiple rounds/phases of a survey by common columns</em></h3><p>Now, we want to merge the six SHED files and combine them into one grand dataset. We call this new dataset <strong>shed_data</strong>.</p><p>But how exactly do we do it? 🤔 Here is a plan: 👇🏾</p><ol><li>Create a <strong>Year</strong> variable in each dataset and put the name of the relevant year in it. We are doing this so that once we merge these 6 datasets, we can identify the respondents in different years.</li><li>Create a <strong>shed_weight </strong>variable in each dataset and put the weight variables in each SHED dataset inside it. Why do we need this? One annoying thing is that different SHED datasets have different names for these weight variables. We are creating a common weight variable to generate weighted statistics.</li><li>Identify the columns that are common across the 6 datasets.</li><li>Merge the 6 datasets using these common columns.</li></ol><h4>(1) Create a year variable (i.e., column) in each dataset</h4><pre>#Create a Year variable<br>shed2017$Year &lt;- 2017<br>shed2018$Year &lt;- 2018<br>shed2019$Year &lt;- 2019<br>shed2020$Year &lt;- 2020<br>shed2021$Year &lt;- 2021<br>shed2022$Year &lt;- 2022</pre><h4>(2) Create a common weight variable in each dataset</h4><pre>#Create a common weight variable (shed_weight) based on the most commonly used weights in each SHED<br>shed2022$shed_weight &lt;- shed2022$weight_pop<br>shed2021$shed_weight &lt;- shed2021$weight_pop<br>shed2020$shed_weight &lt;- shed2020$weight_pop<br>shed2019$shed_weight &lt;- shed2019$weight_pop<br>shed2018$shed_weight &lt;- shed2018$weight2b<br>shed2017$shed_weight &lt;- shed2017$weight3b</pre><h4><strong>(3) Identify the common columns</strong></h4><p>To identify the common columns, run the following code:</p><pre>#Identify the common columns across the 6 datasets<br>common_col_names &lt;- Reduce(intersect, list(names(shed2017),<br>                                           names(shed2018),<br>                                           names(shed2019),<br>                                           names(shed2020),<br>                                           names(shed2021),<br>                                           names(shed2022)))</pre><p>The above code may seem complicated, but do not worry at all!</p><p>It uses four functions: <strong>Reduce( )</strong>, <strong>intersect( )</strong>, <strong>list ( )</strong>, and <strong>names( )</strong>.</p><p>You can easily use the same code for other survey datasets. All you need to change is the name of the datasets inside the <strong>names( )</strong> function.</p><h4>(4) Merge by the common columns</h4><p>We use the <strong>rbind( )</strong> and <strong>select ( )</strong> functions from the<strong> dplyr</strong> package and merge the 6 datasets by running the following code:</p><pre>install.packages(&#39;dplyr&#39;)<br><br>library(dplyr)<br><br>#Merging all 4 by the common columns<br>shed_data &lt;- rbind(shed2017 %&gt;% select(common_col_names),<br>                   shed2018 %&gt;% select(common_col_names),<br>                   shed2019 %&gt;% select(common_col_names),<br>                   shed2020 %&gt;% select(common_col_names),<br>                   shed2021 %&gt;% select(common_col_names),<br>                   shed2022 %&gt;% select(common_col_names))</pre><p>This is what I get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*8lfh4rVHfw9t6rss8CJ9Hw.png" /></figure><p>So, now we have the 6 original SHED datasets and a merged dataset.</p><p>One final thing. The problem is that in the merged dataset, some weights are <em>NA</em> (i.e., missing). We want to eliminate these cases because, without doing so, we cannot generate weighted statistics.</p><pre>nrow(shed_data %&gt;% filter(is.na(shed_weight))) #this shows the total NAs<br><br>shed_data &lt;- shed_data %&gt;% filter(!is.na(shed_weight)) #this eliminates the NAs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*hGmvCzxWPTdoIHw4FV9_1Q.png" /></figure><p>So, we lost 260 respondents out of 71,125 respondents. I suppose that is fine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*nejBUzdUQRmTX9QlMLSKsA.png" /></figure><h3><strong><em>Task 3:</em></strong><em> Creating a line chart</em></h3><p>Now that we have the merged dataset, we want to create a line chart that shows how people’s perceptions of the state of the economy varied between 2017 and 2022.</p><p>First, let’s understand the <strong>B7_b</strong> variable by finding it in the <a href="https://www.federalreserve.gov/consumerscommunities/files/SHED_2022codebook.pdf"><strong>codebook</strong></a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*0XnQl53cL71qP1bu8PZceQ.png" /></figure><p>Let’s look at the distribution of this variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*LtKpI04MGkU5UdUnDtp2_w.png" /></figure><p>So, the <strong>B7_b </strong>is a categorical variable with 5 levels (Excellent, Good, Only fair, Poor, and Refused). Let’s keep the first four categories as the last one (i.e., Refused) is not useful for our analysis.</p><pre>shed_data &lt;- shed_data %&gt;% <br>             filter(B7_b==&quot;Excellent&quot;|B7_b==&quot;Good&quot;|B7_b==&quot;Only fair&quot;|B7_b==&quot;Poor&quot;)</pre><p>Let’s look at the distribution one more time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*X5YsZ04Ek1vBknDzCiea8A.png" /></figure><p>Now, let’s create a binary/dichotomous variable (<strong>economy_poor</strong>) from this categorical variable (<strong>B7_b</strong>) using the <strong>ifelse( )</strong> function. This <strong>economy_poor</strong> variable takes a value of 1 if <strong>B7_b ==</strong> “Poor” and a 0 otherwise.</p><pre>#create a dummy indicating whether the respondent thinks the economic conditions are poor<br>shed_data$economy_poor &lt;- ifelse(shed_data$B7_b==&quot;Poor&quot;,1,0)</pre><p>Let’s see the distribution of the <strong>economy_poor</strong> variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*inL35Y4ohOoYvnJQtU8okg.png" /></figure><p>Now, let’s create a table that shows the proportion of respondents who perceived the economic condition to be poor in each year (2017–2022).</p><pre>#create a table of the proportion of respondents saying the economic conditions are poor<br>shed_data %&gt;% group_by(Year) %&gt;%<br>  summarise(Poor=round(mean(economy_poor)*100,digits=2))</pre><p>The above code returns the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/1*_FPGn7WSc7q_83gBxGHyUg.png" /></figure><p>Let’s use this table to create our basic line chart:</p><pre>install.packages(&#39;ggplot2&#39;)<br><br>library(ggplot2)<br><br>#create a line chart <br>shed_data %&gt;% group_by(Year) %&gt;%<br>  summarise(Poor=round(weighted.mean(economy_poor,shed_weight)*100,digits=2)) %&gt;%<br>  ggplot(aes(y=Poor, x=factor(Year),group=1)) + <br>  geom_line(size=2)+labs(title =&quot;% of Americans Saying Current Economic Conditions Are Poor&quot;,y=&quot;Percent&quot;,x=&quot;&quot;)+<br>  theme_bw()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*kXHSy8M7uK4_H4DjbW_xkQ.png" /></figure><p>And we have our basic line chart!</p><h3><strong><em>Task 4:</em></strong><em> Estimating an empirical model using linear regression</em></h3><p>This is perhaps fairly straightforward!</p><p>Let’s estimate a model that predicts the probability that a respondent says the current economic conditions are poor.</p><p>We use linear regression for the estimation process, using the <strong>ppethm</strong> and <strong>Year</strong> variables as predictors.</p><p>(<em>Of course, you can add many more predictors. I am showing you a simpler example)</em>.</p><p>I use the <strong>feols ( )</strong> function from the <strong>fixest</strong> package. I prefer this function for multiple reasons, including the ease of using heteroskedasticity-robust standard errors and presenting results in tables.</p><p>I estimate two models and present them side-by-side:</p><p><strong>Model 1:</strong> Poor = b0 + b1*ppethm + b2*Year + e</p><p><strong>Model 2:</strong> Poor = b0 + b1*ppethm + b2*Year + b3*ppethm*year + e</p><pre>install.packages(&#39;fixest&#39;)<br><br>library(fixest)<br><br>#estimate models using weighted least squares<br><br>model1 &lt;- feols (economy_poor~factor(ppethm)+factor(Year), data=shed_data, weight=shed_data$shed_weight)<br><br>model2 &lt;- feols (economy_poor~factor(ppethm)+factor(Year)+factor(ppethm):factor(Year), data=shed_data, weight=shed_data$shed_weight, se=&quot;hetero&quot;) #using heteroskedasticity-robust standard error<br><br>etable(model1,model2,digits = 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*f_XwWILsX3tGMnZss8PoVA.png" /></figure><h3><strong><em>Task 5: </em></strong><em>Saving graphs and tables</em></h3><p>Let’s save the above regression table as a .csv file in the folder where the R code and the SHED data files are located.</p><p>We do the following:</p><ol><li>Save the regression table as a dataframe (let’s name it <strong>reg_table</strong>)</li><li>Use the <strong>write.csv( ) </strong>function to export the file</li></ol><p>Let’s run the following code:</p><pre>reg_table&lt;- etable(model1,model2)<br><br>write.csv(reg_table,&quot;reg_table.csv&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2EDDy5VVkcbEy6OM_yw6A.png" /></figure><p>As we can see, the <strong>reg_table.csv </strong>file has been exported to my working folder.</p><p>To export the line chart we created earlier, we use the <strong>ggsave( ) </strong>function immediately after the code used to generate the graph.</p><pre>shed_data %&gt;% group_by(Year) %&gt;%<br>  summarise(Poor=round(weighted.mean(economy_poor,shed_weight)*100,digits=2)) %&gt;%<br>  ggplot(aes(y=Poor, x=factor(Year),group=1)) + <br>  geom_line(size=2)+labs(title =&quot;% of Americans Saying Current Economic Conditions Are Poor&quot;,y=&quot;Percent&quot;,x=&quot;&quot;)+<br>  theme_bw()<br><br>ggsave(&quot;economy_poor_line_chart.png&quot;, width = 13, height = 9, units = &quot;in&quot;)</pre><p>And, just like the table, the figure has been exported to my working folder:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vH-M2pNl0jEFswOFUhyS0A.png" /></figure><h3>Extra Task: <em>Creating a grouped line chart</em></h3><p>Let’s pretend that you would like to investigate how the percentage of people saying the economic conditions are poor varies between <em>Non-Hispanic White</em> respondents and <em>Other</em> respondents.</p><p>In SHED, the <strong>ppethm variable</strong> shows people’s racial and ethnic identities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*M5EQ9APAXbrlMTSuqzibsQ.png" /></figure><p>Let’s make a <strong>non_hispanic_white</strong> dummy variable using the <strong>ifelse( )</strong> function:</p><pre>#create a Non-Hispanic White dummy<br>shed_data$non_hispanic_white&lt;- ifelse(shed_data$ppethm==&quot;White, Non-Hispanic&quot;,&quot;Non-Hispanic White&quot;, &quot;Other&quot;)</pre><p>Now, let’s create a line chart with two lines (one for<em> Non-Hispanic White</em> respondents and one for the <em>Other</em> respondents):</p><pre>#create a line chart<br>shed_data %&gt;% group_by(Year,non_hispanic_white) %&gt;%<br>  summarise(Poor=round(weighted.mean(economy_poor,shed_weight)*100,digits=2)) %&gt;%<br>  ggplot(aes(y=Poor, x=factor(Year),group=factor(non_hispanic_white),linetype=factor(non_hispanic_white))) + <br>  geom_line(size=2)+<br>  labs(title =&quot;% of Americans Saying Current Economic Conditions Are Poor&quot;,y=&quot;Percent&quot;,x=&quot;&quot;,linetype=&quot;Group&quot;)+<br>  theme(axis.text.x = element_text(size=15),<br>        panel.background=element_blank(),<br>        plot.title    = element_text(face = &quot;bold&quot;,   size = 25, hjust = 0.5),<br>        axis.text.y   = element_text(size=15),<br>        axis.title.x   = element_text(size=15),<br>        axis.title.y  = element_text(size=15),<br>        legend.text   = element_text(size=15),<br>        legend.position = &quot;top&quot;,<br>        legend.title=element_text(size=15))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XEFYj1ADBRl_R7D9nAnreQ.png" /></figure><p>Thank you so much for following along 😊</p><p>Here is the link to the third and final part of this crash course:</p><p><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-3-4048a5d98dd4">Analyzing Survey Data in R: A Crash Course (Part 3)</a></p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03da4ec119ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Survey Data in R: A Crash Course (Part 1)]]></title>
            <link>https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-1-9dfa4b110115?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dfa4b110115</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 02:42:20 GMT</pubDate>
            <atom:updated>2025-10-23T22:30:23.006Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step lesson</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HREGJS8a0R3_rlDi" /><figcaption>Photo by <a href="https://unsplash.com/@prateekkatyal?utm_source=medium&amp;utm_medium=referral">Prateek Katyal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This crash course discusses <strong>15</strong> <strong>common tasks for survey data analysis in R</strong>.</p><p>I explain these tasks in three parts. Throughout this course, I use publicly available real-world datasets. This course is most suitable for beginners who want to quickly learn coding in R to begin survey data analysis for their research.</p><blockquote><em>**You can find the video lectures of this crash course on the course website</em>😊<em>**</em></blockquote><p><a href="https://www.vivekanandadas.com/surveycourse.html">Survey Data Analysis in R</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9biGaJwR0ik%3Fstart%3D12%26feature%3Doembed%26start%3D12&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9biGaJwR0ik&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9biGaJwR0ik%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href">https://medium.com/media/56d32b725897adcaed86d9bad377abf2/href</a></iframe><p>I assume you know some fundamentals of R. For example, you know the following operators:</p><blockquote><em>&lt;- (assignment)</em></blockquote><blockquote><em>= (assignment)</em></blockquote><blockquote><em>== (equal)</em></blockquote><blockquote><em>&gt; (greater than)</em></blockquote><blockquote><em>&lt; (less than)</em></blockquote><blockquote><em>&gt;= (greater than or equal)</em></blockquote><blockquote><em>&lt;= (less than or equal)</em></blockquote><blockquote><em>!= (not equal)</em></blockquote><blockquote><em>| (or)</em></blockquote><blockquote><em>&amp; (and)</em></blockquote><h3>Before we get started</h3><p>I assume you have RStudio installed on your computer; if not, please use the <a href="https://login.rstudio.cloud/login?redirect=%2F">cloud version of RStudio</a>.</p><p>This article discusses the following:</p><blockquote><strong><em>Task 1:</em></strong><em> Importing a .csv file directly from the web</em></blockquote><blockquote><strong><em>Task 2:</em></strong><em> Creating a subset (Selecting specific rows)</em></blockquote><blockquote><strong><em>Task 3: </em></strong><em>Selecting specific columns</em></blockquote><blockquote><strong><em>Task 4:</em></strong><em> Creating a binary (a.k.a. dummy) variable</em></blockquote><blockquote><strong><em>Task 5:</em></strong><em> Renaming the levels of a categorical variable</em></blockquote><blockquote><strong><em>Task 6:</em></strong><em> Creating a new categorical variable</em></blockquote><blockquote><strong><em>Task 7:</em></strong><em> Creating a summary statistics table</em></blockquote><blockquote><strong><em>Task 8:</em></strong><em> Creating a barplot</em></blockquote><blockquote><strong>Extra Task: </strong><em>Creating a grouped/stacked barplot</em></blockquote><h3>Task 1: Importing a .csv file directly from the web</h3><p>I use data from the <a href="https://www.consumerfinance.gov/data-research/financial-well-being-survey-data/">National Financial Well-Being Survey</a>, conducted by the Consumer Financial Protection Bureau.</p><p>To download any .csv file openly available on the internet, we can use the <strong>read.csv( )</strong> function. We do two things here:</p><ol><li>Put the URL inside the <strong>read.csv ( ) </strong>function.</li><li>Name the dataframe. For example, I named it <strong>data</strong>.</li></ol><pre>data &lt;- read.csv(&quot;https://www.consumerfinance.gov/documents/5614/NFWBS_PUF_2016_data.csv&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*V0WSVteEZCeUa1zAUBBp2A.png" /></figure><h3>Task 2: Creating a subset (Selecting specific rows)</h3><p>Oftentimes, the target population of our research interest is a subgroup, rather than the whole population.</p><p>For example, let’s pretend we are interested in households with less than $50,000 annual household income. In other words, we want to analyze data from only those respondents who lived in a household with less than $50,000 annual income.</p><p>First, I check the survey <a href="https://files.consumerfinance.gov/f/documents/cfpb_nfwbs-puf-codebook.pdf"><strong>codebook</strong></a> to see how the income variable is coded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRovOrLwPN6rSqJUfAPdQQ.png" /></figure><p>We observe two things. First, the income variable has been named <strong>PPINCIMP</strong>. Second, selecting levels 1, 2, 3, and 4 would provide us with the responses for the respondents living in households with less than $50,000 annual income.</p><p>To get this subset, we use the <strong>filter( )</strong> function from the <strong>dplyr </strong>package. Let’s install the package, load it, and run the code:</p><pre>install.packages(&quot;dplyr&quot;) #Installs the package<br><br>library(dplyr) #Loads the package<br><br>income50k &lt;- data %&gt;% filter(PPINCIMP&lt;=4) #Gets the &lt;$50k income subset</pre><p>Once you run the above code, you will see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*8j7HzaDrvho9m69IzbzOfg.png" /></figure><p>So, now, we have the main dataframe (<strong>data</strong>) and the subset that has only lower-income respondents (<strong>income50k</strong>).</p><p>You can find the income categories inside this new dataframe:</p><pre>table(income50k$PPINCIMP)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*N65efaI7vhwNHgJ3RNQm2Q.png" /></figure><p>The <strong>income50k</strong> dataset only has respondents with less than $50,000 annual household income.</p><h3>Task 3: <em>Selecting specific columns</em></h3><p>Sometimes, we may want to reduce the number of columns in our dataset. In other words, to explore our research questions, we may need only certain variables.</p><p>For example, let’s pretend that in the <strong>income50k </strong>dataset, the variables that we are interested in are: <strong>PPGENDER</strong>, <strong>PPHHSIZE</strong>, <strong>PPINCIMP</strong>, <strong>FWBscore</strong>, and <strong>finalwt</strong>.</p><p>To select the above columns/variables, we can do the following:</p><pre>#Select specific columns in R<br>income50k &lt;- income50k %&gt;% select(PPGENDER,PPHHSIZE,PPINCIMP,FWBscore,finalwt)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*A_oeje2UaQ-Db5fIzKrjzQ.png" /></figure><h3>Task 4: Creating a binary (a.k.a dummy) variable</h3><p>Let’s pretend, we are not interested in all the education categories that our dataset provides. Rather, we want to know something about <em>people with less than a college degree</em> and <em>people with a college degree or above</em>.</p><p>In other words, we are thinking about a binary education variable that takes a value of 1 if a respondent has a<em> college degree or above</em> and 0 if their educational attainment is<em> less than a college degree</em>.</p><p>Let’s find the levels of the <strong>PPEDUC</strong> variable:</p><pre>table(data$PPEDUC)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*DERDJASR9Bey8Z0svUAyew.png" /></figure><p>Let’s understand what these numbers refer to based on the information provided in the codebook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjLg8DUw0OFUXEfEAas5FA.png" /></figure><p>Okay, here is the plan:</p><ol><li>Name the new variable <strong>COLLEGE</strong>.</li><li><strong>COLLEGE</strong> takes a value of 1 if <strong>PPEDUC</strong> &gt;= 4; <strong>COLLEGE</strong> takes a value of 0 if <strong>PPEDUC</strong> &lt; 4.</li></ol><p>Let’s use the <strong>ifelse( )</strong> function to create this new variable:</p><pre>data$COLLEGE&lt;- ifelse(data$PPEDUC&gt;=4,1,0)</pre><p>Let’s see the levels of <strong>COLLEGE</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*-_IFp3IyS6RiBiaNw2PEnA.png" /></figure><p>Awesome! 😊</p><h3>Task 5: Renaming the levels of a categorical variable</h3><p>Let’s check the <strong>PPINCIMP</strong> variable’s coding one more time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*xpB1JUW-PwWecSsFqsXRyw.png" /></figure><p>Note that the levels of the <strong>PPINCIMP </strong>variable are 1, 2, …, and 9, which are not very helpful. Therefore, we would like to rename the levels to make our analyses more comprehensible.</p><p>To rename the levels of this categorical variable, we can use the <strong>recode( ) </strong>function.</p><p>Here is the plan:</p><ol><li>Inside the <strong>recode</strong>( ) function, we first enter the name of the data frame (<strong>data</strong>) and then the name of the column (<strong>PPINCIMP</strong>). This is what we have on the left-hand side of the assignment operator (&lt;-) as well. We are doing this because we want to rename the levels of the <strong>PPINCIMP</strong> variable, which is located inside the <strong>data</strong> dataframe as a column.</li><li>Next, we follow something like this: “Old name of a level” = “New name of the level”. And we do the same for each level.</li></ol><pre>data$PPINCIMP &lt;-  recode(data$PPINCIMP,&quot;1&quot;=&quot;Less than $20,000&quot;, <br>                         &quot;2&quot;=&quot;$20,000 to $29,999&quot;,<br>                         &quot;3&quot;=&quot;$30,000 to $39,999&quot;,<br>                         &quot;4&quot;=&quot;$40,000 to $49,999&quot;,<br>                         &quot;5&quot;=&quot;$50,000 to $59,999&quot;,<br>                         &quot;6&quot;=&quot;$60,000 to $74,999&quot;,<br>                         &quot;7&quot;=&quot;$75,000 to $99,999&quot;,<br>                         &quot;8&quot;=&quot;$100,000 to $149,999&quot;,<br>                         &quot;9&quot;=&quot;$150,000 or more&quot;)<br><br>table(data$PPINCIMP)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UlQs3MMPFi-HVQXrr2lpg.png" /></figure><p>It worked! 😊</p><h3>Task 6: Creating a new categorical variable</h3><p>Now, let’s pretend we are interested in people at the intersection of their gender identity and the generation that they belong to.</p><p>Let’s have a quick look at these variables.</p><pre>table(data$PPINCIMP)<br>table(data$PPGENDER)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*KW2mLV7lx1NdcZVnIcBQgg.png" /></figure><p>We observe that <strong>generation </strong>has four levels. Let’s check the codebook to learn what these numbers mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njhGuLrVemiayJBe70Gb9w.png" /></figure><p>Again, let’s check the codebook to learn what the levels of <strong>PPGENDER </strong>mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gw1VvTCi8yUaHQQP3VI8Q.png" /></figure><p>Okay, so the <strong>generation</strong> variable has 4 levels and the <strong>PPGENDER</strong> variable has 2 levels. At the intersection of the two variables, there will be 4 x 2 = 8 levels.</p><p>However, the dataset does not provide us with a separate variable with these 8 levels. Let’s create it ourselves!</p><p>This time, I use the <strong>ifelse ( ) </strong>function.</p><p><em>**Note that the following code may seem a bit challenging. But nothing to worry about! You do not have to memorize anything. Simply copy and paste the following code, adjust the variable names according to your dataset, and you are all set!**</em></p><pre>data$GENERATION.GENDER &lt;- ifelse(data$PPGENDER==1 &amp; data$generation==1, &#39;Male, Pre-Boomer&#39;,<br>                                           ifelse(data$PPGENDER==1 &amp; data$generation==2, &#39;Male, Boomer&#39;,<br>                                                  ifelse(data$PPGENDER==1 &amp; data$generation==3, &#39;Male, Gen X&#39;,<br>                                                         ifelse(data$PPGENDER==1 &amp; data$generation==4, &#39;Male, Millennial&#39;,<br>                                                                ifelse(data$PPGENDER==2 &amp; data$generation==1, &#39;Female, Pre-Boomer&#39;,<br>                                                                       ifelse(data$PPGENDER==2 &amp; data$generation==2, &#39;Female, Boomer&#39;,<br>                                                                              ifelse(data$PPGENDER==2 &amp; data$generation==3, &#39;Female, Gen X&#39;,<br>                                                                                                                   &#39;Female, Millennial&#39;)))))))</pre><p>Here is a summary of my code shown above:</p><ol><li>I created a new variable named <strong>GENERATION.GENDER </strong>(that is why I have <strong>data$GENERATION.GENDER</strong> at the beginning of the code).</li><li>Because we have 8 categories at the intersection of the two variables (i.e., <strong>generation</strong> and <strong>PPGENDER</strong>), we write <strong>ifelse( )</strong> conditions 7 times.</li><li>The final category does not require a condition, as respondents who do not match any of the 7 conditions (referring to the 7 categories) will be assigned to this category, which is “Female, Millennial”.</li></ol><p>Let’s check the levels of the <strong>GENERATION.GENDER</strong> variable:</p><pre>table(data$GENERATION.GENDER)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vumxxayz75IMP2qDvZ1s8A.png" /></figure><h3>Task 7: Creating a summary statistics table</h3><p>Let’s pretend we want to know how financial well-being — indicated by a variable(<strong>FWBScore</strong>) on a scale ranging from 0 to 100 — differs at the intersection of generation and gender.</p><p>Here is the plan:</p><ol><li>Use the <strong>group_by( )</strong> function from the <strong>dplyr </strong>package. We want to group respondents based on the levels of the <strong>GENERATION.GENDER</strong> variable we created.</li><li>We want these statistics for each level of <strong>GENERATION.GENDER</strong>: the number of respondents, mean FWBscore, median FWBscore, and standard deviation of FWBscore. We use these functions: <strong>n( ) → </strong>number of people<strong>; mean( ) → </strong>mean<strong>; median ( ) → </strong>median<strong>; sd( ) → </strong>standard deviation<strong>.</strong></li><li>We use the <strong>summarise( )</strong> function to get the above summary statistics.</li><li>We use the <strong>round( )</strong> function to round numbers to one place after the decimal.</li><li>We put the table inside a dataframe called <strong>table.</strong></li></ol><p>Here is the code:</p><pre>table &lt;- data %&gt;% group_by(GENERATION.GENDER) %&gt;%<br>  summarise(Count=n(), <br>            Mean.FWBScore=round(mean(FWBscore),digits=1), <br>            Median.FWBScore=round(median(FWBscore),digits=1),<br>            SD.FWBScore=round(sd(FWBscore),digits=1)<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*B-gHy6gR8HJi5IKafQa9TA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*GzCZyQqP_j67eI1tObXewA.png" /></figure><p>It seems that the oldest male respondents are the most financially satisfied, while the youngest female respondents are the least financially satisfied. Apparently, the intersection matters.</p><p>A better way to present the above table would be to show the same in a barplot.</p><p>Let’s create a barplot!</p><h3>Task 8: Creating a barplot</h3><p>We are going to use the table we created earlier to make a barplot.</p><p>Here is the plan:</p><ol><li>Use the <strong>ggplot( )</strong> function from the <strong>ggplot2</strong> package.</li><li>Inside the <strong>ggplot ( )</strong> function, first, put the name of the dataframe. The name of the dataframe here is <strong>table </strong>(as I named it earlier)<strong>.</strong></li><li>Inside the <strong>aes( )</strong> function, after <strong>“x = ”</strong>, put the value of <em>the variable based on which your groups are categorized</em>. In this case, the variable is <strong>GENERATION.GENDER</strong>, and therefore, I type <strong>x=GENERATION.GENDER</strong>. Similarly, after “<strong>y= </strong>”, put the value of the <em>outcome variable</em>. In this case, the variable is <strong>Mean.FWBScore.</strong></li><li>Inside the <strong>geom_bar ( )</strong> function, type <strong>stat= “identity”</strong>.</li></ol><pre>install.packages(&quot;ggplot2&quot;) #Installs the package<br><br>library(ggplot2)<br><br>library(dplyr) #Loads the package<br><br>ggplot(table, aes(x=GENERATION.GENDER, y=Mean.FWBScore)) + <br>  geom_bar(stat = &quot;identity&quot;)</pre><p>Okay, here is how my basic barplot looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgc1DoL04iu2_ZAGRmrjLA.png" /></figure><p>I want to improve it by doing the following:</p><ol><li>Flip the graph using the <strong>coord_flip( )</strong> function.</li><li>Change the theme to the <strong>theme_light( )</strong> function.</li><li>Change the y-axis label.</li><li>No need for the x-axis label as the categories are self-explanatory.</li></ol><p>I do the above by running the following code:</p><pre>ggplot(table, aes(x=GENERATION.GENDER, y=Mean.FWBScore)) + <br>  geom_bar(stat = &quot;identity&quot;)+ <br>  coord_flip()+<br>  theme_light()+<br>  labs(y=&quot;Average Financial Well-Being Score&quot;, x=&quot; &quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*zvzcXajZijNnsitp0s95-A.png" /></figure><p>Looks fairly decent!</p><h3>Extra Task: Creating a grouped/stacked barplot</h3><p>Sometimes, we want to create a grouped or a stacked barplot.</p><p>For example, in the barplot we created above, we may want to incorporate the household income (<strong>PPINCIMP</strong>) variable into the analysis. To do that, first, we need to incorporate <strong>PPINCIMP</strong> into the table:</p><pre>table &lt;- data %&gt;% group_by(GENERATION.GENDER,PPINCIMP) %&gt;%<br>  summarise(Count=n(), <br>            Mean.FWBScore=round(mean(FWBscore),digits=1), <br>            Median.FWBScore=round(median(FWBscore),digits=1),<br>            SD.FWBScore=round(sd(FWBscore),digits=1)<br>  )</pre><p>Next, we do the following:</p><pre>ggplot(table, aes(x=GENERATION.GENDER, y=Mean.FWBScore,fill=factor(PPINCIMP))) +<br>  geom_bar(stat=&quot;identity&quot;,position = &quot;dodge&quot;)+<br>  coord_flip()+<br>  labs(x=&quot; &quot;,y=&quot;Average Financial Wellbeing Score&quot;,fill=&quot;Household Income&quot;)+<br>  theme_minimal()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*x5eecVkY1DJyMV8lQltJZw.png" /></figure><p>To convert the above grouped bar plot into a stacked bar plot, we make a small change into the code. We change the <em>position = “dodge”</em> to <em>position = “stack”.</em></p><pre>ggplot(table, aes(x=GENERATION.GENDER, y=Mean.FWBScore,fill=factor(PPINCIMP))) +<br>  geom_bar(stat=&quot;identity&quot;,position = &quot;stack&quot;)+<br>  coord_flip()+<br>  labs(x=&quot; &quot;,y=&quot;Average Financial Wellbeing Score&quot;,fill=&quot;Household Income&quot;)+<br>  theme_minimal()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ecgPIUIEOUnOiRiCOQvOyA.png" /></figure><p>The stacked version of the barplot is harder to understand. And I would prefer presenting the grouped barplot in the context of our analysis.</p><p>Before we end, you may have noticed that the income categories are not properly ordered. This is because the levels of factors are, by default, organized <em>alphanumerically</em>.</p><p>Below, I present the code to recreate the group barplot with income categories shown from lowest to highest:</p><pre>#Reording the levels of the PPINCIMP variable<br>table &lt;- table %&gt;% mutate(PPINCIMP = factor(PPINCIMP, levels = c(&quot;Less than $20,000&quot;,<br>                                                        &quot;$20,000 to $29,999&quot;,<br>                                                        &quot;$30,000 to $39,999&quot;,<br>                                                        &quot;$40,000 to $49,999&quot;,<br>                                                        &quot;$50,000 to $59,999&quot;,<br>                                                        &quot;$60,000 to $74,999&quot;,<br>                                                        &quot;$75,000 to $99,999&quot;,<br>                                                        &quot;$100,000 to $149,999&quot;,<br>                                                        &quot;$150,000 or more&quot;)))<br><br>#Creating a grouped barplot<br>ggplot(table, aes(x=GENERATION.GENDER, y=Mean.FWBScore,fill=PPINCIMP)) +<br>  geom_bar(stat=&quot;identity&quot;,position = &quot;dodge&quot;)+<br>  coord_flip()+<br>  labs(x=&quot; &quot;,y=&quot;Average Financial Wellbeing Score&quot;,fill=&quot;Household Income&quot;)+<br>  theme_minimal()</pre><p>Also, I change two more things. First, I use different colors than the ones that we got as default. Second, I reorder the categories inside the GENERATION.GENDER variable. Here is the code to do both:</p><pre># Define custom colors from light gray to dark green<br>income_colors &lt;- c(&quot;#E8F5E9&quot;, &quot;#C8E6C9&quot;, &quot;#A5D6A7&quot;, &quot;#81C784&quot;, &quot;#66BB6A&quot;, <br>                   &quot;#4CAF50&quot;, &quot;#388E3C&quot;, &quot;#2E7D32&quot;, &quot;#1B5E20&quot;)<br><br># Define the order of generations, from youngest to oldest<br>generation_order &lt;- c(&quot;Male, Millennial&quot;, &quot;Female, Millennial&quot;, &quot;Male, Gen X&quot;, &quot;Female, Gen X&quot;, <br>                      &quot;Male, Pre-Boomer&quot;, &quot;Female, Pre-Boomer&quot;,&quot;Male, Boomer&quot;, &quot;Female, Boomer&quot;)<br><br># Reorganize the levels of GENERATION.GENDER to reflect the desired order<br>table$GENERATION.GENDER &lt;- factor(table$GENERATION.GENDER, levels = generation_order)<br><br># Create the plot<br>ggplot(table, aes(x = GENERATION.GENDER, y = Mean.FWBScore, fill = PPINCIMP)) +<br>  geom_bar(stat = &quot;identity&quot;, position = &quot;dodge&quot;) +<br>  coord_flip() +<br>  labs(x = &quot; &quot;, y = &quot;Average Financial Wellbeing Score&quot;, fill = &quot;Household Income&quot;) +<br>  theme_minimal() +<br>  scale_fill_manual(values = income_colors)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7nwJd-uf7YeBZtNEpJeYQ.png" /></figure><p>A “smarter” way (i.e., utilizing the pipe operator from the dplyr package strategically) to write the above can be the following:</p><pre># Reorganize the levels of PPINCIMP and GENERATION.GENDER, and plot the graph in one step<br>table %&gt;%<br>  mutate(<br>    PPINCIMP = factor(PPINCIMP, levels = c(&quot;Less than $20,000&quot;, &quot;$20,000 to $29,999&quot;, &quot;$30,000 to $39,999&quot;,<br>                                           &quot;$40,000 to $49,999&quot;, &quot;$50,000 to $59,999&quot;, &quot;$60,000 to $74,999&quot;, <br>                                           &quot;$75,000 to $99,999&quot;, &quot;$100,000 to $149,999&quot;, &quot;$150,000 or more&quot;)),<br>    GENERATION.GENDER = factor(GENERATION.GENDER, levels = c(&quot;Male, Millennial&quot;, &quot;Female, Millennial&quot;, <br>                                                             &quot;Male, Gen X&quot;, &quot;Female, Gen X&quot;, <br>                                                             &quot;Male, Pre-Boomer&quot;, &quot;Female, Pre-Boomer&quot;,<br>                                                             &quot;Male, Boomer&quot;, &quot;Female, Boomer&quot;))<br>  ) %&gt;%<br>  ggplot(aes(x = GENERATION.GENDER, y = Mean.FWBScore, fill = PPINCIMP)) +<br>  geom_bar(stat = &quot;identity&quot;, position = &quot;dodge&quot;) +<br>  coord_flip() +<br>  labs(x = &quot; &quot;, y = &quot;Average Financial Wellbeing Score&quot;, fill = &quot;Household Income&quot;) +<br>  theme_minimal() +<br>  scale_fill_manual(values = c(&quot;#E8F5E9&quot;, &quot;#C8E6C9&quot;, &quot;#A5D6A7&quot;, &quot;#81C784&quot;, &quot;#66BB6A&quot;, <br>                               &quot;#4CAF50&quot;, &quot;#388E3C&quot;, &quot;#2E7D32&quot;, &quot;#1B5E20&quot;))</pre><p>This brings me to the end of the first part of this course.</p><p>Here are the links to the second and third parts:</p><ul><li><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-2-03da4ec119ec">Analyzing Survey Data in R: A Crash Course (Part 2)</a></li><li><a href="https://vivdas.medium.com/analyzing-survey-data-in-r-a-crash-course-part-3-4048a5d98dd4">Analyzing Survey Data in R: A Crash Course (Part 3)</a></li></ul><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dfa4b110115" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Reasons Why Statistics is So Hard to Learn]]></title>
            <link>https://vivdas.medium.com/5-reasons-why-statistics-is-so-hard-to-learn-460c350fd0?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/460c350fd0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Sat, 31 Dec 2022 06:20:02 GMT</pubDate>
            <atom:updated>2025-10-23T22:30:45.832Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lrSoS4JcewFIMIYq" /><figcaption>Photo by <a href="https://unsplash.com/@helloimnik?utm_source=medium&amp;utm_medium=referral">Nik</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every year, thousands of students around the globe — some willingly, some unwillingly — enroll in introductory statistics, data analytics, consumer analytics, and many other related courses. And it is not difficult to find students who would put these courses on top of their scariest academic experiences.</p><p>In this article, I try to take the perspective of the typical student enrolled in an introductory statistics course and explain five reasons why we struggle to grasp the course’s core content.</p><h3>1. Randomness is unintuitive</h3><p>The <a href="https://dictionary.cambridge.org/us/dictionary/english/random">Cambridge Dictionary</a> defines <strong>random</strong> as “<em>happening, done, or chosen by chance rather than according to a plan</em>.”</p><p>As a human being, I do not possess an<em> </em>intuitive<em> </em>understanding of random events. I love to pretend things happen for a reason. Otherwise, how do I make my existence meaningful?</p><p>Think about sports. Suppose the captain of a sports team, especially in cricket, loses several coin tosses in a row. Many supporters then start blaming the captain as if it is their fault. Similarly, if several fifty-fifty decisions go against the team you support, you may begin to suspect a conspiracy and blame the referee or umpire. In both situations, although the outcomes are driven by randomness and not by any deliberate purpose, people often do not perceive these outcomes as random events.</p><p>As statistics deals with randomness, much of the subject matter is often unintuitive.</p><p>Before I took my first statistics course, had you asked me, “How would you divide people into two groups which, on average, should be identical?” I would have said, “I would gather their information, start looking for pairs of people who are identical to each other, assign them to the two groups, and keep repeating the process.”</p><p>And then I took statistics and learned that the most effective way would be to rely on random assignment (e.g., coin flips)! 🤦🏾‍♂️</p><p>Indeed, <em>the power of randomness to create presumably organized outcomes</em> is hard to fathom. Instructors should spend more time building a solid understanding of this crucial concept.</p><h3><strong>2. Common sense does not work</strong></h3><p>Let’s say your instructor is teaching you how to interpret the results of a t-test.</p><p>In the example, they are trying to investigate whether there is a difference in annual income between two groups of people. After running the test, they conclude:</p><p>“The difference in the sample average annual income between the two groups is $100. However, this difference is not significantly different from 0 at the 5% level.”</p><p>Statements such as the above one can be challenging for a beginner.</p><p>At first, the instructor says the difference in average income is $100, and then they say this $100 difference is not different from 0. The beginner wonders, how can the value of something simultaneously be $100 and not different from zero? 🤔</p><p>Unfortunately, there is no easy way to correctly explain statistical jargon. I realized it while struggling to grasp them as a student and even more while trying to explain them in my previous articles.</p><ul><li><a href="https://vivdas.medium.com/statistical-significance-are-you-interpreting-correctly-7fff26130fb7">Statistical Significance: Are You Interpreting Correctly?</a></li><li><a href="https://towardsdatascience.com/confidence-interval-are-you-interpreting-correctly-a8834ba5a99b">Confidence Interval: Are You Interpreting Correctly?</a></li><li><a href="https://towardsdatascience.com/why-you-should-prefer-confidence-interval-over-p-value-e32293bd174c">Why You Should Prefer Confidence Interval over p-value</a></li></ul><h3>3. Statistical tests are “acausal”</h3><p>The key purpose of statistical inference is to explore the value of an unknown population parameter based on a sample estimate.</p><p>On the contrary, <em>causal inference </em>requires <em>counterfactual reasoning.</em></p><p>I explain the fundamentals of causal inference in the following article:</p><p><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-1d5f0e70a778">How to Explore the Effect of Doing Something? (Part 1)</a></p><p>The problem is that statistical tests — such as t-test, chi-squared test, F-test, etc. — are done for the purpose of statistical inference and not for causal inference. Whether you can make valid causal claims based on the findings of your statistical tests depends on the <em>data-generating process </em>and the<em> model you estimate</em>.</p><p>Many instructors do not sufficiently explain this key issue. Consequently, students start making causal interpretations based on a finding’s statistical significance (i.e., whether the p-value is &lt; 0.05).</p><p>For example, beginners often struggle to detect the difference between the following statements:</p><ol><li>A one-unit change in X is positively associated with 2.3 units change in Y</li><li>A one-unit increase in X causes/leads to/generates 2.3 units increase in Y</li></ol><p>Some instructors do mention that <em>“correlation does not imply causation”</em> but do not explain the concept well enough. Fortunately, these concepts can be fairly easily explained using some simple directed acyclic graphs.</p><p>In the context of regression analysis, I describe some of these concepts in the following articles:</p><ul><li><a href="https://vivdas.medium.com/confounding-variable-and-spurious-correlation-key-challenge-in-making-causal-inference-4e33d8ba60c2">Confounding Variable and Spurious Correlation: Key Challenge in making Causal Inference</a></li><li><a href="https://vivdas.medium.com/regression-and-causal-inference-which-variables-should-be-added-to-the-model-fd95a759f78">Regression and Causal Inference: Which Variables Should Be Added to the Model?</a></li></ul><h3>4. Lost in formulae and software functions</h3><p>As a student, I attended multiple graduate-level statistics courses in which many of my classmates were lost in mathematical formulae and running statistical tests in R. Also, as a teaching assistant in an undergraduate-level consumer analytics course, I observed many students getting lost in Excel functions. Consequently, many students cannot focus on internalizing statistical reasoning, which should be the primary learning outcome in any introductory statistics course.</p><p>Of course, students need certain mathematical and software skills for a deeper understanding of statistics. Nevertheless, the core ideas of <em>statistical reasoning </em>can be explained using only toy examples.</p><p>Instructors should ensure that students spend most of their time building a solid understanding of fundamental concepts, rather than worrying about how to manually calculate the standard error of a coefficient under time pressure during an exam.</p><h3>5. Fake datasets do not inspire</h3><p>In many courses (especially in online ones), instructors generate random numbers from statistical distributions to create variables and use these fake variables to explain how to conduct statistical tests.</p><p>I understand the value of this approach. First, generating fake datasets can help create simple examples. Second, the process is convenient because the instructor does not have to search for a publicly available dataset, which can be messy due to missing data. Third, the instructor, along with teaching a statistical method, can teach Monte Carlo simulation. Fourth, the steps of statistical tests do not change depending on the data-generating process. So, one may argue that there should be no differential effect on student learning depending on using fake datasets vs. real-world datasets.</p><p>However, in the context of introductory statistics courses, I argue that using real-world datasets, alongside fake datasets for toy examples, can enhance the typical student’s learning.</p><p>Personally, I feel more empowered when I learn how to conduct a statistical test using a real-world dataset. Why? Because as a student of applied social sciences, I enrolled in a statistics course not out of love for statistical distributions and tests, but to learn methods that help me investigate real-world phenomena. Working with a real-world dataset makes me feel like I am progressing in the right direction.</p><h3>Wrapping Up</h3><p>Regardless of whether we become data analysts, since all of us are consumers of data products, a solid understanding of statistical reasoning can help us become critical thinkers and better decision-makers.</p><p>The introductory statistics course is the first and last time many students receive training on describing and analyzing data. Consequently, instructors of these courses should prioritize the following:</p><ol><li>Appreciate the fact that the concepts that seem obvious to them (e.g., consequences of random sampling and assignment, the central limit theorem, etc.) may not seem obvious to a beginner.</li><li>Help students build a solid understanding of statistical reasoning using toy examples.</li><li>Save students from getting lost in formulae and software functions.</li><li>Use graphs to explain under what circumstances statistical association between two variables can be interpreted as a causal effect.</li><li>Empower and inspire students by teaching statistical analyses using real-world datasets.</li></ol><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=460c350fd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Data in Social Sciences vs. Engineering: Key Differences to Keep In Mind]]></title>
            <link>https://vivdas.medium.com/analyzing-data-in-engineering-vs-social-sciences-key-differences-to-keep-in-mind-4a306aff168c?source=rss-964a6be9969------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a306aff168c</guid>
            <category><![CDATA[social-science]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Vivekananda Das]]></dc:creator>
            <pubDate>Sun, 25 Dec 2022 03:59:16 GMT</pubDate>
            <atom:updated>2025-10-23T22:31:05.130Z</atom:updated>
            <content:encoded><![CDATA[<h3>Analyzing Data in Social Science vs Engineering: Key Differences to Keep In Mind</h3><p>This week, as I am wrapping up another semester in the applied social sciences, I have been revisiting some of the key challenges I faced while transitioning from engineering to social science.</p><p>In this article, I discuss some key differences between typical engineering and social data and how these differences require different quantitative modeling and interpretation approaches.</p><h3><strong>Engineering Data</strong></h3><p><em>***Before we begin: neither all engineering data are experimental, nor all social data are non-experimental. Also, engineers do not always deal with data that follow from a law/theory. I am using simple (and perhaps too stereotypical) examples here for the sake of explaining the key points of this article***</em></p><p>Let’s begin with <strong>Ohm’s law</strong>, a fundamental physics law many of us explored through experiments in high school. Let’s revisit the law:</p><blockquote><strong><em>Ohm’s law</em></strong><em> states that the current through a conductor between two points is directly proportional to the voltage across the two points.</em></blockquote><blockquote><em>Introducing the constant of proportionality, the resistance, one arrives at the usual mathematical equation that describes this relationship:</em></blockquote><blockquote>I = V/R</blockquote><blockquote><em>where </em>I<em> is the current through the conductor,</em> V<em> is the voltage measured across the conductor, and </em>R<em> is the resistance of the conductor</em></blockquote><blockquote>(source: <a href="https://en.wikipedia.org/wiki/Ohm%27s_law">Wikipedia</a>)</blockquote><p>Here are a few things to note from a data analyst’s perspective:</p><ol><li>The<strong> unit of analysis </strong>is “one conductor.”</li><li>When you are running the <strong>experiment</strong>, you are changing voltage across two points of the <strong>same conductor.</strong></li><li><strong>The conductor remains the same </strong>throughout the experiment.</li><li>You change voltage <strong>(independent/explanatory/treatment variable).</strong></li><li>You observe a change in the flow of current <strong>(dependent/outcome variable).</strong></li><li>You (perhaps unconsciously)<em> </em>assume that, for this conductor, at a particular voltage (e.g., 0.8 V), the observed current (e.g., 0.38 A) would be the same regardless of when you measure it. To elaborate on this, let’s say, you ran an experiment in which you gradually increased voltage in this sequence: 0V, 1V, and 2V, and observed current in this sequence: 0A, 0.49A, and 0.98A. You are assuming that had you run the experiment in the reverse order, for example, in this sequence: 2V, 1V, and 0V, you would have observed current in this sequence 0.98A, 0.49A, and 0A. To summarize, you are assuming that there are pairs of voltage and current — for example, (OV and 0A), (1V and 0.49A), (2V and 0.98A), etc. — that do not change depending on the sequence of experimentation.</li></ol><p>Next, you create a plot like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_R_PU5uZssw1zPh7D1JlQ.png" /></figure><p>Several observations:</p><ol><li>Because these data are experimental, you definitely know <strong>the treatment</strong> <strong>variable </strong>(i.e., voltage) and <strong>the outcome</strong> <strong>variable </strong>(i.e., current).<strong> </strong>By <strong>treatment variable</strong>, we refer to the <strong>cause, </strong>and by <strong>outcome variable</strong>, we refer to the <strong>effect.</strong></li><li>Working with data like the one from Ohm’s law is a great privilege. You are not stuck in a situation where you have to ponder over <em>whether X causes Y, </em>or <em>Y causes X,</em> or <em>both X and Y are caused by Z.</em> Here, you clearly know that <em>the change in voltage is the cause (treatment), and the change in current is the effect (outcome).</em></li><li>You can make a <strong>counterfactual claim</strong>, such as <em>“Had I not changed the voltage, the current would not have changed.”</em></li><li>Also, you can make a <strong>causal claim</strong>, such as <em>“A change in voltage across the conductor causes a change in current through the conductor.”</em></li><li>There is reasonable <strong>homogeneity, </strong>i.e., whether you run the experiment in the U.S. in 1923 or Bangladesh in 2023, for the same type of conductor, you will get the same numbers, and they will fall on a line, ignoring negligible noise introduced by measurement errors, power loss, etc.</li></ol><p>For this conductor, you can describe the relationship between current and voltage as follows:</p><p><em>I = 0.49 * V</em></p><p>So, your key conclusion would be: with a one-unit increase in voltage across the conductor, the current through the conductor increases by 0.49 units.</p><h4>Note that we already had a law (I=V/R), which we tested using empirical data. Put differently, in this case, we already knew the data-generating process (as described by Ohm’s law).</h4><h4>Unfortunately, given the enormous complexity of the social world, often, there is not a robust law or theory that we can test using empirical data. In other words, unless we are running a randomized experiment in a controlled environment, we rarely know the data-generating process.</h4><h4>Often, using empirical data, we try to understand social laws and theories.</h4><h4>To make things even more complicated, these social laws and theories can change across time and societies and require frequent updates.</h4><h3>Modeling Like an Engineer, but the Data Are Social</h3><p>Not all engineering datasets are as smooth as Ohm’s law data. However, generally, they tend to be less messy than social data, which facilitates our modeling of the relationships among variables using mathematical formulae.</p><p>Sometimes, while modeling relationships among variables using social data, someone trained as an engineer may disregard the underlying data-generating processes and focus too much on curve fitting (i.e., developing a mathematical expression that best fits the data).</p><p>Importantly, social data tend to be non-experimental more often than engineering data. Curve fitting practices that work well for experimental data may not be suitable for non-experimental data, especially when proposing a change in policy or treatment to achieve a desirable outcome.</p><p>Let’s pretend I am working with a dataset that has data on the <em>monthly spending by customers at a store (in $) </em>and their <em>satisfaction with the store (measured on a scale from 1 to 10)</em>. I create a scatter plot that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*G1gNQfZlpWNWVFbqO54IZA.png" /></figure><p>This plot is not quite as smooth as the Ohm’s law one. But we understand humans are not quite as homogeneous as conductors, right? So, we do not get too disheartened by the relatively more messiness in social data (i.e., not all dots falling on a line).</p><p>From the plot, it appears that a straight line (with an intercept) may be a pretty good fit for the data, as customer satisfaction tends to increase (almost) linearly as monthly spending increases.</p><blockquote>Are you observing any similarity between the spending-satisfaction relationship and the voltage-current relationship? 🤔</blockquote><p>I propose the following model to mathematically express the relationship:</p><blockquote>Customer Satisfaction = 3.33 + 0.01*Monthly Spending</blockquote><p>However, you propose a revision to the model: a quadratic term should be added. I accept your proposal and update the model as follows:</p><blockquote>Customer Satisfaction = 3.2 + 0.01*Monthly Spending-0.0001*(Monthly Spending)²</blockquote><p>Maybe someone else proposes another mathematical model that fits the data even better. But wait for a second!</p><p>Let’s stop worrying about mathematics for a bit and compare Ohm’s law data and the customer data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*XkgE8s3R3lqIFVCA6ynhdg.png" /></figure><p>Given the above differences, it may be inappropriate to interpret the findings of your model the same way you interpret the findings of your model in the Ohm’s Law case.</p><p>For example, solely based on the model you have, the following counterfactual and causal claims are possibly incorrect:</p><p><em>“Had the less-spending customers spent more, their satisfaction would have been higher”</em></p><p><em>“An increase in monthly spending causes an increase in customer satisfaction”</em></p><p>The correct interpretations are:</p><ol><li><em>Customers who spend more tend to be more satisfied</em></li><li><em>Monthly spending is positively associated/correlated with customer satisfaction</em></li></ol><p>Although the two interpretations mentioned above are <em>technically correct</em>, they are <em>not helpful</em> for your manager who assigned you the data analysis responsibility to determine possible ways to increase customer satisfaction.</p><p>The interpretations are wishy-washy because you just described a <em>passive observation,</em> whereas they need an <em>active policy solution</em>.</p><p>The problem here is more than just about proposing a mathematical model that fits the data; it requires <em>identifying a policy change</em> that would increase customer satisfaction.</p><h3>You Need a Theory of How the World Works</h3><p>Let’s pretend the process that generates our customer data can be drawn as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NWfOVD1bG9kku7lKXuf4aQ.png" /></figure><p>So here is the explanation of the data-generating process:</p><p><em>“Higher income customers tend to spend more at the store. Also, higher income customers tend to be more satisfied with the store (maybe because the store is high-end).”</em></p><p>Because I disregarded the data-generating process in the following model,</p><blockquote>Customer Satisfaction = 3.33 + 0.01*Monthly Spending</blockquote><p>I suggested that a $1 increase in customer spending would increase customer satisfaction by 0.01 points.</p><p>This suggestion may not be helpful from a policy perspective, though. For example, if the store manager raises product prices to make customers spend more, their satisfaction may actually go down!</p><p>In the above example, we observed a case in which we can predict the value of an outcome variable based on the value of an explanatory variable. Let’s call this a <em>factual prediction</em>.</p><p>However, for policy advice, we need a counterfactual prediction <em>(i.e., we need to predict what would have happened had we actively changed the value of something, keeping all else constant).</em> Unfortunately, despite being pretty good at factual prediction, our model is not helpful for counterfactual prediction.</p><p>How do we deal with this issue? The general solution is to figure out the common causes of the treatment and outcome variables and add them to your model as additional explanatory variables. I describe this briefly in another article.</p><p><a href="https://towardsdatascience.com/what-does-it-mean-to-control-for-something-in-multiple-regression-744880620988">What Does It Mean to Control for a Variable in Regression?</a></p><h3>Curve Fitting is Extremely Useful, Though</h3><p>If you have read this far, you may wonder why almost all data science education focuses on curve fitting if these practices may not be helpful for policy advice.</p><p>In many real-world applications, all you need to do is propose a model that can reliably predict the value of some <strong><em>Y</em></strong> based on the values of some <strong><em>X</em></strong><em>s</em>.</p><p>Let’s pretend you work for an insurance company. The insurer wants to determine which potential insurees will most likely go bankrupt. This is a scenario in which you may be happy with a model that can make factual predictions.</p><p>Assume that you have a dataset that has only two variables. You know the number of traffic violations someone has committed in the previous year and whether they are bankrupt. Based on the two variables, you propose the following model:</p><blockquote>Bankrupt = 0.05 + 0.1*Number of Traffic Violations</blockquote><p>Let’s pretend that using the above model, you can reasonably predict the probability of an insuree going bankrupt.</p><p>How about proposing a policy solution based on the above model? It appears that the probability of bankruptcy would be lowest if we set the number of traffic violations to 0. However, it may not be a piece of great policy advice if we propose that by pardoning all traffic violations (i.e., setting the value to 0), we can drastically reduce the incidence of bankruptcy.</p><h3>Wrapping Up</h3><p>I wish someone had suggested the following at the beginning of my data science education.</p><p>Before modeling any data,</p><blockquote><em>First, ruminating over the data-generating process is a great idea.</em></blockquote><blockquote><em>Second, we should ask ourselves why we are analyzing the data. Are we just trying to predict the value of an outcome variable as a passive observer, or are we trying to propose a piece of policy advice to an active policymaker?</em></blockquote><p>Given that we clearly understand these two issues, only and only<em> </em>then should we think about the appropriate mathematical modeling.</p><p><em>** If you found this article helpful, please consider following me! Also, please consider becoming an Email Subscriber so that you receive an email next time I post something!**</em></p><ul><li><a href="https://vivdas.medium.com/regression-and-causal-inference-which-variables-should-be-added-to-the-model-fd95a759f78">Regression and Causal Inference: Which Variables Should Be Added to The Model?</a></li><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-1d5f0e70a778">How to Explore the Effect of Doing Something? (Part 1)</a></li><li><a href="https://vivdas.medium.com/how-to-explore-the-effect-of-doing-something-part-2-228f857bb060">How to Explore the Effect of Doing Something? (Part 2)</a></li><li><a href="https://vivdas.medium.com/why-is-correlation-neither-necessary-nor-sufficient-for-causation-in-non-experimental-data-900c866d59d7">Why Is Correlation Neither Necessary Nor Sufficient for Causation (in Non-Experimental Data)?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a306aff168c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>