<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mona Khalil on Medium]]></title>
        <description><![CDATA[Stories by Mona Khalil on Medium]]></description>
        <link>https://medium.com/@mona_kay?source=rss-8a78d77bcd92------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*z3sz-tghXcuuBtV7w_eDjg.png</url>
            <title>Stories by Mona Khalil on Medium</title>
            <link>https://medium.com/@mona_kay?source=rss-8a78d77bcd92------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:16:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mona_kay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Effective use of Demographic Information]]></title>
            <link>https://medium.com/in-the-weeds/effective-use-of-demographic-information-e2b61a90f38?source=rss-8a78d77bcd92------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2b61a90f38</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[research-methods]]></category>
            <category><![CDATA[conference]]></category>
            <dc:creator><![CDATA[Mona Khalil]]></dc:creator>
            <pubDate>Mon, 06 Apr 2020 20:43:45 GMT</pubDate>
            <atom:updated>2020-04-06T20:43:45.236Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>A Recap from Speaking at the NYC School of Data</strong></h3><p>Demographic data plays a major part in a large proportion of research projects, machine learning models, and scientific findings. Ideas about people based on their race and gender permeate all areas of society and can inform our decisions in ways we aren’t even aware of. Many of these models and projects are later found to produce <a href="https://www.nytimes.com/2019/12/06/business/algorithm-bias-fix.html">biased results</a>, doing harm to already marginalized groups in society.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VnGPf7u3aisrJzsJ3vuLDg.png" /></figure><p>My colleague Devin Johnson and I recently presented on this topic at the annual <strong>NYC School of Data, </strong>introducing an aspect of data science ethics we’ve been investigating for years. We prepared for a casual session with 10 to 15 people…and instead packed the room to its limits with 50+ attendees! We generated incredible conversations as we shared ideas with our colleagues, and were met with wide enthusiasm for this topic everywhere we went. We’re excited today to share the top 3 problems we identified in our talk, and 3 recommendations for approaching demographic data analysis in your work.</p><h3>Problems with Demographic Data Collection and Analysis</h3><p><strong>The importance of demographic information in a specific analysis is rarely questioned.</strong> The inclusion of demographic questions assessing study participants’ race, gender, age, and income will rarely receive pushback from research teams and ethics committees. Asking a research team why they need to collect demographic information returns nearly identical responses — “how else will I compare groups?”, or “that’s how I identify group differences”. The question of why these categories are important is rarely asked. It’s just inferred.</p><p><strong>Commonly used demographic groups (i.e., race/ethnicity, income categories) differ widely throughout history and across countries.</strong> Race/ethnicity categories commonly used in research tend to be based on those used in the <a href="https://www.census.gov/topics/population/race/about.html">U.S. Census</a>, which assume differences between people based on country of origin and/or observable characteristics. Categories differ across countries that are otherwise culturally similar (e.g., Canada), and instead tend to reflect the relationship between a majority group and multiple minority groups.</p><p><strong>Statistical tests comparing large numbers of subgroups in a dataset are prone to Type 1 (false positive) errors.</strong> The more groups you compare using methods such as a t-test or ANOVA, the more likely you are to yield results that are false positives. Even with statistical tests that make adjustments for multiple groups, each additional comparison adds to the likelihood that your results are a fluke. Additionally, the further you subset your dataset, the smaller the sample, thus adding to this likelihood.</p><h3><strong>What can I do instead?</strong></h3><p>We proposed a set of ideas involving the use of demographic information in designing research projects and analyzing data. In our previous work, we’ve found that these approaches have been effective at generating useful conversations with stakeholders, refocusing our analytics efforts, and making more concrete recommendations about the populations we were studying.</p><p><strong>Don’t collect demographic information.</strong> Simply put, unless your hypotheses directly involve the needs of a specific demographic group. Are you studying the impact of a health and wellbeing program on a specific immigrant population in New York City? Go for it. Are you conducting a study to understand how people learn at different ages? Maybe don’t collect data on race.</p><p><strong>Have demographic information already? Ignore it. </strong>Before exploring demographic data, consider what hypotheses you would come up with if you did not have that data available. What other sources of data would you collect or find in order to add value to your work? What assumptions would you make? If you’re studying topics such as drug usage among teenagers, try looking at data about each student’s family income, relationship to parents, relationship to peers, and drug usage among adults in the region. Many studies looking at drug and alcohol usage focus on poor, minority students, even though research shows that affluent students use drugs and alcohol at similar if not higher rates[1]. The routine focus on demographic data in this area of research detracts from the larger scale of the problem and a comprehensive solution.</p><p><strong>Can’t get out of analyzing demographic data? Use fewer categories.</strong> If you or your stakeholders are used to expecting results comparing demographic groups, consider reframing the way in which you break down your categories. Which demographic group is the <em>dominant</em> group (i.e., the group you suspect has an advantage) in the context of your work? Which group(s) do you expect are at a disadvantage? For example, if you are examining pay equity by race and gender, you may want to compare <strong>male vs. non-male (women, non-binary people) employees </strong>and <strong>white vs. non-white employees</strong>. This allows groups in small minorities (i.e., the two non-binary employees) to retain anonymity in your analysis and evaluates groups in accordance with power structures. It also reduces the number of group comparisons, minimizing your chance of false-positive differences in your statistical tests.</p><h3><strong>Concluding Thoughts</strong></h3><p>We expect that these recommendations are only the beginning of a longer conversation about the use of demographic data in analytics and data science. If you have thoughts on more sophisticated ways to make use of demographic information, I invite you to reach out. A concerted effort to evaluate our approaches will help us understand populations, subpopulations, and develop more robust research methods using this information in the coming years.</p><h3>Resources</h3><p>[1] Luthar, S. S., &amp; Latendresse, S. J. (2005). Children of the affluent: Challenges to well-being. <em>Current directions in psychological science</em>, <em>14</em>(1), 49–53.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2b61a90f38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/effective-use-of-demographic-information-e2b61a90f38">Effective use of Demographic Information</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Greenhouse’s Scorecard Rating System]]></title>
            <link>https://medium.com/in-the-weeds/greenhouses-scorecard-rating-system-d66eb16d4306?source=rss-8a78d77bcd92------2</link>
            <guid isPermaLink="false">https://medium.com/p/d66eb16d4306</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hiring]]></category>
            <dc:creator><![CDATA[Mona Khalil]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 22:08:22 GMT</pubDate>
            <atom:updated>2019-11-05T15:39:27.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VaqfyO3n5JIYIww1tLJeQw.png" /></figure><p>If you’ve ever seen or filled out a scorecard in Greenhouse Recruiting, you likely noticed our unique measurement scale for assessing candidate attributes and performance. We’re regularly asked about our rationale behind a combination of colors and emojis instead of a traditional 5-point scale (i.e., 1 to 5, Strongly Disagree to Strongly Agree), and met with a variety of feedback on our choice.</p><p>Our choice of measurement scale here was far from random — in fact, our choice was heavily informed by research in the field of psychometrics. <strong>Psychometrics</strong> is the study of quantitative measurement practices in the social sciences. A psychometrician generally researches best practices in evaluating the quality of metrics (i.e., survey items), measurement scales (i.e., Strongly Disagree to Strongly Disagree), and other related factors that contribute to the accuracy of the behavior or process you’re trying to capture [1].</p><p>Throughout this article, we’ll introduce our justification for excluding numbers, text/worded scales, and our ultimate choice to use color &amp; symbolic scales in our product. We’ll also share some interesting facts about how people tend to respond to surveys with only the smallest tweaks in measurement scales.</p><p>—</p><p><strong>Showing numbers on a scale can significantly impact how people respond to your questions.</strong></p><p>Have you noticed the wide variety of numerical scales used in survey questions to represent the <em>same set </em>of response choices? The following 5-point Likert scale is a common set of choices available as a range of answers to a survey question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbR8Fefvgqy3-rvlJopQhQ.png" /></figure><p>And so is this variation of the same scale:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zx6VtT0O1gbnNVLJ57eevQ.png" /></figure><p>Several studies on the use of Likert scales have demonstrated that people’s responses will vary based on the numbers shown (or not shown) to survey participants. [2] Survey participants tend to evaluate &amp; assign different weights to numbers that may introduce bias in any survey scale. Traditional academic research tends to collect data from larger sample sizes (i.e., 200–300 respondents), helps balance out the variation in response rates from people’s interpretation of a scale. That’s generally much higher than you’ll see in the hiring process — as a hiring manager, you’ll likely receive candidate ratings from 5 to 10 people, and even fewer responses on each of the individual attributes to be evaluated. Any step we can take to reduce error and bias improves the quality of your candidate evaluations.</p><p><strong>There are strong cultural and regional differences in how people choose to respond to worded measurement scales.</strong></p><p>Did you know that residents of the United States tend to respond overwhelmingly positive to most survey questions they receive than many other parts of the world? [3] These differences in survey responses don’t reflect any cultural differences in optimism or agreement — in fact, people in the United States tend to be <em>less</em> positive and trusting of institutions as a whole. [4] In the U.S. respondents likely respond to a question as <em>Strongly Agree</em> unless they have a clear reason to disagree with the statement being made. In other parts of the world, such as in mainland China [3], respondents will respond more neutrally unless they have a clear justification for strongly agreeing with the statement.</p><p>Excluding a worded scale ensures that your hiring managers can be confident that your candidate is being evaluated consistently across interviewers, and is not subject to bias associated with different individuals’ interpretations of <em>Strongly Agree/Strongly Disagree.</em></p><p><strong>People tend to respond more consistently and powerfully to color heuristics than they do to words.</strong></p><p>Humans process colors &amp; symbols in a different region of the brain than words or numbers. [5] We also more readily associate colors with a negative or positive valuation based on opacity, etc. [6] By using shades of red, yellow, and green as heuristics to evaluate your candidates, we hope to provide you with more consistent results across individuals and teams as you make your hiring decisions.</p><p><strong>Conclusion</strong></p><p>Ultimately, our scorecard rating setup was created in the interests of providing customers with the best possible platform for reducing bias and individual differences. We’re always open to feedback and evidence-based approaches to better reaching that goal.</p><p><strong>References</strong></p><p>[1] Psychometric Society: <em>What is psychometrics?</em> <a href="https://www.psychometricsociety.org/content/what-psychometrics">https://www.psychometricsociety.org/content/what-psychometrics</a></p><p>[2] Weijters, B., Cabooter, E., &amp; Schillewaert, N. (2010). The effect of rating scale format on response styles: The number of response categories and response category labels. <a href="https://doi.org/10.1016/j.ijresmar.2010.02.004">https://doi.org/10.1016/j.ijresmar.2010.02.004</a></p><p>[3] Lee, J. W., Jones, P. S., Mineyama, Y., &amp; Zhang, X. E. (2002). Cultural differences in responses to a Likert scale. <em>Research in nursing &amp; health</em>, <em>25</em>(4), 295–306.<strong> </strong><a href="https://doi.org/10.1002/nur.10041">https://doi.org/10.1002/nur.10041</a></p><p>[4] Twenge, J. M., Campbell, W. K., &amp; Carter, N. T. (2014). Declines in trust in others and confidence in institutions among American adults and late adolescents, 1972–2012. <em>Psychological Science</em>, <em>25</em>(10), 1914–1923. <a href="https://doi.org/10.1177%2F0956797614545133">https://doi.org/10.1177/0956797614545133</a></p><p>[5] Peterson, Bradley S., et al. “An fMRI study of Stroop word-color interference: evidence for cingulate subregions subserving multiple distributed attentional systems.” <em>Biological psychiatry</em> 45.10 (1999): 1237–1258. <a href="https://doi.org/10.1016/S0006-3223(99)00056-6">https://doi.org/10.1016/S0006-3223(99)00056-6</a></p><p>[6] Piotrowski, C., &amp; Armstrong, T. (2012). Color Red: Implications for applied psychology and marketing research. <em>Psychology and education-An Interdisciplinary Journal</em>, <em>49</em>, 55–57.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d66eb16d4306" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/greenhouses-scorecard-rating-system-d66eb16d4306">Greenhouse’s Scorecard Rating System</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>