<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Martha McRoy on Medium]]></title>
        <description><![CDATA[Stories by Martha McRoy on Medium]]></description>
        <link>https://medium.com/@mmcroy?source=rss-847fef297fb5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oLjDFI6Q5Z_qmfa79JqarA.jpeg</url>
            <title>Stories by Martha McRoy on Medium</title>
            <link>https://medium.com/@mmcroy?source=rss-847fef297fb5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:51:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mmcroy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How survey mode affects Americans’ views of international affairs]]></title>
            <link>https://medium.com/pew-research-center-decoded/how-survey-mode-affects-americans-views-of-international-affairs-da3a540c6ddc?source=rss-847fef297fb5------2</link>
            <guid isPermaLink="false">https://medium.com/p/da3a540c6ddc</guid>
            <category><![CDATA[public-opinion-research]]></category>
            <category><![CDATA[mode-effects]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[polling]]></category>
            <category><![CDATA[survey-mode]]></category>
            <dc:creator><![CDATA[Martha McRoy]]></dc:creator>
            <pubDate>Wed, 21 Apr 2021 14:58:50 GMT</pubDate>
            <atom:updated>2021-04-21T14:58:50.885Z</atom:updated>
            <content:encoded><![CDATA[<h3>Examining how survey mode affects Americans’ views of international affairs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDni4qXchv6VFIuyV4cKpA.png" /><figcaption>(Pew Research Center illustration)</figcaption></figure><p>Over the past several years, Pew Research Center has <a href="https://www.pewresearch.org/fact-tank/2019/02/27/what-our-transition-to-online-polling-means-for-decades-of-phone-survey-trends/">migrated much of its U.S. polling</a> from phone to its online <a href="https://www.pewresearch.org/methods/2019/02/27/growing-and-improving-pew-research-centers-american-trends-panel/">American Trends Panel</a> (ATP). Data collection for the U.S. component of the annual <a href="https://www.pewresearch.org/global/">Global Attitudes Survey</a> — a major, cross-national study typically fielded in dozens of countries via face-to-face and phone interviews — followed suit <a href="https://www.pewresearch.org/politics/2021/02/24/majority-of-americans-confident-in-bidens-handling-of-foreign-policy-as-term-begins/">this year</a>.</p><p>Mindful that mode differences can affect response patterns as well as analysis of findings, we conducted a “bridge” survey in 2020, fielding the same Global Attitudes questionnaire online and by phone. While the Center has previously examined the implications of <a href="https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/">migrating online</a> for a variety of mostly domestically oriented survey items, this was our first extensive test of how mode might affect how Americans answer foreign policy and internationally focused questions.</p><p>Our bridge survey included a total of 78 substantive opinion questions. In this post, we’ll summarize the range of percentage-point differences across all items but concentrate our analysis on response patterns to four-point scales — the most common question format on the Global Attitudes Survey. Specifically, we’ll examine how key four-point scale questions performed in terms of direction and intensity of responses, as well as rates of item non-response. Our findings underscore the importance of examining all three elements when interpreting questions and trends that have migrated from phone to online.</p><p>We should note at the outset that we confirmed that the samples for the phone and online components of the bridge survey were comparable and that the online sample matched the characteristics of a typical ATP survey wave.</p><h3>Differences in attitudes by phone vs. online</h3><p>The chart below summarizes the percentage-point difference between phone and online for each of the 78 survey items included in the bridge survey. Only the single largest difference by response categories for any given question is reported. For example, let’s say 20% of our phone respondents said “yes” to a particular question, 15% said “no” and 65% said “maybe.” And let’s say 30% of our online respondents said “yes” to the same question, 10% said “no” and 60% said “maybe.” In this hypothetical, we would report the 10-point difference for the “yes” category since it’s the largest of the three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-bdxZ0MCKcKxaL0LEra0w.png" /></figure><p>Generally, to achieve statistical significance at the 95% confidence level, a difference of 4 points or higher is needed. When we look at the largest differences by response category, 73% of questions (57 of 78) met this threshold, including 19 survey items in which the margin of difference was 10 points or more.</p><p>Two-thirds of these double-digit differences were associated with items that employed a four-point scale. These differences by mode will factor in how we analyze and report changes in Global Attitudes findings in the U.S. To inform that analysis, we took a closer look at response dynamics <em>across</em> categories, focusing on key trend items that employ four-point scales.</p><h3>Trend questions, four-point scales and mode effects</h3><p>As mentioned above, four-point scale questions were the most common question type on the Global Attitudes bridge survey (34 items, or 44% of all substantive opinion questions) and also the question type associated with some of the largest percentage-point differences across modes. While these overall findings are instructive, we sought to better understand the response dynamics within individual questions. This meant examining responses across all categories and assessing the implications for analysis and interpretation.</p><p>Four-point scale items allow respondents to express their opinion in terms of direction (e.g., positive versus negative) and intensity (e.g., very, somewhat, not very much or none at all). Some of the Global Attitude Survey’s most consistent trend questions employ four-point scales. For example, confidence in global leaders to do the right thing regarding world affairs is measured using a four-point scale: a lot of confidence, some confidence, not too much confidence or no confidence at all.</p><p>The bridge survey posed this confidence question about six world leaders: German Chancellor Angela Merkel, French President Emanuel Macron, British Prime Minister Boris Johnson, Chinese President Xi Jinping, Russian President Vladimir Putin and then-U.S. President Donald Trump. For all of these leaders except Trump, there were double-digit differences across modes for one response category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*j2eImaFQ_maaR8hXs09oVQ.png" /></figure><p>As the chart reveals, for non-U.S. leaders, the response category “not too much confidence” consistently exhibited the largest percentage-point differences across modes, ranging from 10 points for Putin to 14 points for Macron and Xi. The direction of difference was also consistent: online ATP respondents were more likely than phone respondents to answer “not too much confidence.” Phone respondents, in turn, were consistently more likely to give the extreme response “no confidence at all” when asked about foreign leaders, though the mode differences were smaller, ranging from 2 points for Macron up to 7 points for Xi. (Part of these differences is attributable to higher item nonresponse on the phone, which is described below.)</p><p>The response pattern diverged when Americans were asked about confidence in their own president. No single response category for Trump had a mode difference that exceeded 10 points, and the largest observed difference was in the share of respondents who answered “a lot of confidence.”</p><p>The response patterns by mode for the questions about foreign leaders were fairly common across four-point scale items on the bridge survey. For instance, there were 12 four-point questions with differences of 5 points or more in the extreme negative response category, with 10 of them showing higher rates for phone respondents than ATP panelists.</p><p>To get a better sense for what such mode differences might mean for the analysis of Global Attitudes findings, we took a closer look at response dynamics across all categories of key four-point scale trend items.</p><p>One example was the question about confidence in Chinese president Xi. As noted, ATP panelists were 14 points more likely to say they had “not too much confidence” in Xi compared to those on the phone, while those on the phone were 7 points more likely than online respondents to say they had “no confidence at all.” These are differences in intensity rather than direction of opinion. If we focus only on direction, by recoding the four categories into two (confidence versus no confidence), the single largest category difference by mode is reduced to 7 points (lack of confidence in Xi). The collapse of four-point scales to bi-modal response categories is a standard feature of Global Attitudes reporting. In this case, it highlights a meaningful consistency in the overall level of Americans’ confidence in China’s leader. Of course, the collapse of categories does not eliminate the mode effect, and the interpretation of the trend should take into account possible shifts in intensity of opinion, too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*51hEHdsKnY6kTzQQ_9SJmg.png" /></figure><h3>Item nonresponse for four-point scale items is lower online than phone</h3><p>The rate of item nonresponse is a frequent concern when migrating from interviewer-administered phone surveys to self-administered online polls. The ATP, like many web-based surveys, does not explicitly offer respondents the option of answering “don’t know” or “refuse.” In fact, the way ATP panelists would not respond to a question is by skipping it completely. And while “don’t know” or “refuse” options are not explicitly offered on traditional Global Attitudes phone surveys, respondents nonetheless give these answers with substantial frequency in interviewer-administered environments. The bridge survey confirmed this pattern: Phone respondents were more likely than their ATP counterparts to offer “don’t know” or “refuse” responses.</p><p>Some of the four-point scale questions that had the largest nonresponse differences across modes were the items on confidence in world leaders to do the right thing in world affairs, with four of six items having higher nonresponse by phone than online (ranging from 5 to 12 points). Another set of four-point scale questions with relatively large item nonresponse differences were those about favorability toward other countries and international organizations. Of seven such questions, six had higher nonresponse rates by phone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oYrgEoHM3ATdiPFI0JUwUg.png" /></figure><p>Looking at the example of American attitudes toward NATO, nonresponse rates were 16 points lower on ATP than by phone. This had a substantial impact when coupled with mode differences across the item’s other four response categories, especially “somewhat favorable” and “somewhat unfavorable.” When we collapsed all four categories down to two, this lower item nonresponse rate led to both a more positive <em>and</em> more negative opinion of NATO among ATP respondents (5 percentage points more positive and 10 points more negative). Thus, in the case of the NATO question, the migration from phone to online was associated with a change in <em>two </em>directions, as well as differences in opinion intensity.</p><p>Item nonresponse across modes differed less for the question measuring U.S. favorability than for parallel items about foreign countries and international organizations. The same was true for the question about confidence in Trump, compared with other international leaders. In each case, differences in intensity, more than direction, were evident across modes.</p><h3>Conclusions and further considerations</h3><p>Insights from the Global Attitudes bridge survey in the U.S. suggest that issue salience — that is, how cognitively engaged the respondent is with a particular topic or issue — may play a role in response differences. For instance, we noted more striking mode effects when it came to relatively low-salience questions about foreign leaders as well as other countries and international organizations. By contrast, questions about the U.S. and its president showed more subdued mode differences with lower nonresponse. Managing how best to reliably measure low-salience issues across modes will be an ongoing challenge — especially as engagement with topics, as well as response biases, can vary across countries.</p><p>Another takeaway from this study is that the analysis and interpretation of questions and trends spanning modes benefits from paying attention to both the direction and intensity of opinion, as reflected across response categories. A single question can reveal both consistencies and differences in response patterns as we migrate the Global Attitudes Survey from phone to online. The survey’s tradition of collapsing four-point scales into dichotomous categories in its reporting may prove helpful in tracking high-level trends. At the same time, attention to how the intensity of opinion may be changing will be key to interpreting the strength and momentum of core trends going forward.</p><p><em>Martha McRoy is a research methodologist focusing on international survey research methods at Pew Research Center.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da3a540c6ddc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pew-research-center-decoded/how-survey-mode-affects-americans-views-of-international-affairs-da3a540c6ddc">How survey mode affects Americans’ views of international affairs</a> was originally published in <a href="https://medium.com/pew-research-center-decoded">Pew Research Center: Decoded</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The impact of survey mode on measuring public opinion in Italy]]></title>
            <link>https://medium.com/pew-research-center-decoded/the-impact-of-survey-mode-on-measuring-public-opinion-in-italy-d4d04c3e9b8c?source=rss-847fef297fb5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4d04c3e9b8c</guid>
            <category><![CDATA[survey-mode]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[polling]]></category>
            <category><![CDATA[mode-effects]]></category>
            <category><![CDATA[survey-methodology]]></category>
            <dc:creator><![CDATA[Martha McRoy]]></dc:creator>
            <pubDate>Wed, 31 Mar 2021 17:59:57 GMT</pubDate>
            <atom:updated>2021-03-31T17:59:57.358Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrtrWmpRSLvQ_dKc2eiraA.png" /><figcaption>(Pew Research Center illustration)</figcaption></figure><p>Over the past two decades, Pew Research Center has regularly included Italy as part of its annual, cross-national poll known as the <a href="https://www.pewresearch.org/global/2020/09/15/us-image-plummets-internationally-as-most-say-country-has-handled-coronavirus-badly/">Global Attitudes Survey</a>. Until recently, it was one of the few Western European countries where we continued to rely on face-to-face rather than phone interviews to conduct our nationally representative surveys. We hesitated to switch our data collection in Italy from face-to-face to phone due to concerns about population coverage, potential sample and measurement bias, and overall data quality.</p><p>We had the opportunity in the last few years to rigorously test and evaluate the use of phone survey methods in Italy thanks to two special, ad hoc projects that were not subject to our normally compressed international polling timeline or the constraints of adhering to longstanding trend questions. The first was a national poll about <a href="https://www.pewforum.org/2017/08/31/five-centuries-after-reformation-catholic-protestant-divide-in-western-europe-has-faded/">religious polarization</a>; the second was a nationwide survey on <a href="https://www.journalism.org/2018/05/14/in-western-europe-public-attitudes-toward-news-media-more-divided-by-populist-views-than-left-right-ideology/">media and politics</a>. Together, these projects boosted our confidence in the sample representativeness of phone surveys in Italy. Moreover, they underscored the benefits of computer-assisted telephone interviewing, or CATI, including lower costs, improved quality control and more efficient project management — features we were eager to bring to our annual survey work in Italy. The key question was whether a shift in survey mode would significantly affect responses to core survey items and the analysis of long-established trends.</p><p>To assess <a href="https://www.pewresearch.org/methods/2019/02/07/methods-101-mode-effects/">mode effects</a> in Italy, we conducted a “<a href="http://www.asasrms.org/Proceedings/y2018/files/867003.pdf">bridge survey</a>” in June and July 2019 — simultaneously fielding identical questions and response options via both in-person and telephone interviews. This allowed us to contrast the two approaches and assess the implications for reporting time-series measures, some of which stretch back to 2002, essentially bridging the gap across measurement differences that are attributable to mode. The bridge survey further confirmed that phone surveys can accurately represent Italy’s entire population, but it also revealed differences in how people respond to questions over the phone as opposed to in person.</p><h3>Background: The Global Attitudes Survey</h3><p>Every year since 2002, the Center has fielded the <a href="https://www.pewresearch.org/global/">Global Attitudes Survey</a>, usually in around 20 to 40 countries per year. By design, this strategy allows for comparisons across countries and regions, as well as over time. One challenge in this type of comparative, cross-national analysis is the possibility that attitudes expressed by respondents may reflect elements of a survey’s design rather than actual changes in public opinion. Researchers need to be sensitive to changes in how a survey is administered.</p><p>Traditionally, the Global Attitudes Survey has interviewed Italians using face-to-face methods, with questions asked over time using the same verbiage and fieldwork adhering to similar data-quality standards from one year to the next. This approach gives us confidence that we can attribute differences in public opinion across years to actual changes in attitudes. A switch from asking respondents questions in person to calling them over the phone, however, is a significant shift in methods. This is why we decided to use a bridge survey to estimate what that change might mean for our measures of public opinion.</p><p>Our measures — namely the questions asked as part of the Global Attitudes Survey — generally focus on geopolitics, democracy and globalization, as well as other topical issues such as technology, gender and the media. Salient regional issues also may be included in specific countries to provide timely insights into current events and prominent figures. For example, the Global Attitudes Survey regularly asks people for their confidence in heads of state, but the leaders included in each questionnaire vary by region. In 2019, for example, we asked people in Asian countries about their confidence in Japan’s prime minister, while in Europe we asked about the Hungarian prime minister.</p><h3>Differences in attitudes when asking in person versus by phone</h3><p>Our bridge survey included a total of 79 attitudinal questions. For each question, we analyzed and reported the percentage-point difference between face-to-face and phone estimates. For each question, we used the largest difference across all response options.</p><p>To summarize the findings broadly:</p><p>· Roughly half of the differences between questions (42 items out of 79, or 53%) were 9 points or less when comparing face-to-face and phone results. That included 17 items for which the difference was 5 points or less. (For this survey, achieving statistical significance at the 95% confidence level generally required a 6-point difference between the modes.)</p><p>· Among the 37 items with double-digit differences, 25 questions were 14 points or less.</p><p>· Six questions had differences of 20 points or more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*eQlKy7N9x54CcWZnqqzzJQ.png" /></figure><p>Many of the differences we observed in the bridge survey are related to these two features of the questionnaire’s design and its administration: the frequency of item nonresponse and whether “volunteered” responses were accepted.</p><p>All questions asked as part of the Global Attitudes Survey are voluntary, which means respondents can refuse to answer any item for any reason or say they don’t know enough to provide a response, which is known as item nonresponse. Item nonresponse was more likely to occur in person than on the phone, but in most cases the differences were relatively minor. For about three-quarters of the questions (58 of 79 items, or 73%), the item nonresponse difference between modes was 4 points or less, though there may have been larger differences across the substantive response options. This suggests that attitudinal differences across modes were <em>not</em> simply a function of item nonresponse differences.</p><p>Where we did find large differences in item nonresponse, we checked if these could be associated with significant <em>net </em>directional attitudinal shifts. We looked for a significant difference in only one overall direction — meaning there’s a statistically significant difference in the overall positive direction across modes (e.g., <em>net</em> confidence, approve or yes) but no significant difference in the negative direction (e.g., <em>net</em> no confidence, disapprove or no). We found that a majority of questions with statistically significant differences in nonresponse of 5 points or more (14 of 21 items, or 67%) also had an associated attitudinal difference between modes that was significantly different in one direction only, illustrating that larger rates of nonresponse might lead to meaningful shifts in attitudes for some questions.</p><p>Some questions in the Global Attitudes Survey also allow respondents to provide an answer that might not be explicitly offered by the interviewer. That’s what we refer to as a volunteered response. For instance, when asked to choose between two options, some respondents — usually a small share — may say that both (or neither) options are acceptable to them. In our bridge survey, 20% of the questionnaire (16 items) allowed for these additional volunteered responses. As we saw with item nonresponse, volunteered responses were more common with face-to-face interviews for each of these items, and in some cases these volunteered responses were associated with rather large discrepancies. The two questions with the largest differences across modes, in fact, were found within volunteered response options.</p><p>The following examples are illustrative of these two issues we encountered when comparing face-to-face and phone survey results.</p><h3>Differential item nonresponse: The case of confidence in world leaders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*hZHYn_5-DXR5zqDPN-YSQw.png" /></figure><p>In our spring 2019 questionnaire, we asked people in Italy about their confidence in six heads of state: French President Emmanuel Macron, German Chancellor Angela Merkel, Hungarian Prime Minister Viktor Orbán, Russian President Vladimir Putin, then-U.S. President Donald Trump and Chinese President Xi Jinping. For each leader, face-to-face respondents were more likely than phone respondents not to offer a response. These differences ranged from 4 points for Trump and Merkel to 24 points for Orbán.</p><p>These differences in item nonresponse by mode were sometimes associated with meaningful attitudinal differences, meaning that the <em>net</em> differences in the substantive findings were in one direction. In other cases, differences in item nonresponse did not seem to affect the attitudinal findings.</p><p>In the case of Xi, those interviewed in person were 13 percentage points more likely not to respond (22% vs. 9%), while phone respondents were 13 points more likely to express confidence in him (37% vs. 24%), illustrating a one-sided shift attributable to nonresponse.</p><p>Likewise, with confidence in Putin, face-to-face interviews were 6 points more likely than telephone interviewees not to offer an opinion, while phone respondents were 6 points more likely than in-person respondents to express confidence in the Russian leader (44% vs. 38%).</p><p>Things played out differently in the case of Orbán. Around four-in-ten face-to-face interviewees (39%) did not offer an opinion of the Hungarian prime minister, compared with only 15% of those interviewed by phone. But that 24-point difference was essentially evenly distributed into both <em>confidence</em> and <em>no confidence</em> ratings on the phone, maintaining the same 12-point net difference. In other words, the different levels of item nonresponse did not alter the prime minister’s <em>net</em> confidence rating: It was 37 percentage points both in the in-person interviews (49% no confidence vs. 12% confidence) and in the phone interviews (61% vs. 24%). This equal “redistribution” was the case with Macron as well, though more muted with a 7-point item nonresponse difference.</p><p>We saw generally similar patterns of differential item nonresponse by mode across our polling in other countries in 2019. For example, among all other European countries surveyed for our Global Attitudes Survey, item nonresponse for confidence in Xi ranged from 6 to 12 points for the six countries surveyed via phone, but from 29 to 51 points for the seven face-to-face countries.</p><h3>Volunteered responses: The case of Italy’s relations with the U.S. and Russia</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*4MwNBEuQHsJCiBUi9P0WQw.png" /></figure><p>There were 16 questions in our bridge survey that accepted volunteered responses. The largest substantive difference across modes for five of these questions was the volunteered option, including the single largest difference in our study: a 38-point margin on a new question about Italy’s relationships with the United States and Russia.</p><p>When asking respondents whether it was more important for Italy to have a close relationship with the U.S. or Russia, we accepted two volunteered responses not provided by the interviewer: <em>both relationships are equally important</em> or <em>neither are important</em>. Overall, 49% of those surveyed face-to-face provided one of these volunteered answers while far fewer (14%) did so on the phone.</p><p>The greater use of the volunteered option has a substantial impact on how we understand Italians’ views of their bilateral priorities. In the face-to-face survey, 45% said both relationships are equally important, while 38% said a close relationship with the U.S. is more important. But on the phone, 72% said having a close relationship to the U.S. was more important, and only 7% said both.</p><h3>Understanding our results</h3><p>Overall, the bridge survey provided important insights into what might be expected when migrating from face-to-face to telephone surveys in Italy. The good news is that this project, along with previous Center studies, found sample respondents to be nationally representative across both modes. The more challenging issue was the impact on our measures of Italian public opinion.</p><p>While our experiment was not designed to test the precise causal mechanism driving the mode differences we found, prior <a href="https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/">methodological research</a> on mode effects can help make sense of these results. For instance, there’s some evidence that interviewers may be better able to <a href="https://academic.oup.com/poq/article-abstract/67/1/79/1873914?redirectedFrom=fulltext">build rapport</a> with respondents in person than over the phone. This greater trust and common understanding between respondents and interviewers can help explain the higher rates of item nonresponse and volunteered responses within the face-to-face context we found in our comparison. When speaking in person to an interviewer, respondents may be more comfortable to admit a lack of familiarity with a given subject (as in the case of the Hungarian prime minister) or to provide an answer that is not offered by the interviewer (as in the case of Italy’s relationships with the U.S. and Russia).</p><p><a href="https://pprg.stanford.edu/wp-content/uploads/2003-NES-Mode-Paper.pdf">Other research</a> suggests that respondents on the phone may be more likely to “satisfice” — that is, provide answers in a satisfactory but low-effort way — than during a face-to-face interview, which helps to explain lower rates of nonresponse on the phone but also lower rates of spontaneous, volunteered answers.</p><h3>Conclusion</h3><p>Overall, we found telephone surveys in Italy to be nationally representative and to have numerous benefits over face-to-face surveys, including lower costs, improved quality control and more efficient project management. All these factors can help inform our migration from in-person to telephone interviews in Italy for the Global Attitudes Survey. Transparent reporting of our mode shift, meanwhile, helps educate audiences to the meaning of changes in public opinion — that is, when differences are likely attributable to actual changes in attitudes and when they’re likely attributable to changes in methodology. Our bridge study was central to this goal, showing nontrivial differences by mode. In fact, 47% of our substantive questions had differences of 10 points or more across modes. This required us to think carefully about how we could mitigate differences due to changes in data collection in the future.</p><p>One strategy would be to consider changes to the survey instrument itself, such as reducing or potentially eliminating the number of volunteered responses. This would require recasting questions and response options so that interviewers read all responses previously deemed acceptable as volunteered, essentially eliminating any volunteered responses from the survey itself. This method would require appropriate responses to be known during questionnaire design, which may not always be possible. Of course, changing known volunteered response options from past surveys to explicit response options in future surveys may have unintended consequences which could compromise the over-time comparability.</p><p>In our own work, based on the results of this experiment, we’ve taken steps to reduce the number of questions accepting volunteered responses. In the Global Attitudes Survey we fielded in the summer of 2020, only 4% of items allowed volunteered responses. Likewise, only 4% of the questionnaire for our 2021 survey includes items accepting volunteered responses.</p><p>Whether it’s viable to switch modes in other countries in the future depends on a number of factors in addition to methodological considerations. These include research goals, practical issues of timing and costs, and in-country survey infrastructure and expertise. The <a href="https://www.pewresearch.org/fact-tank/2020/04/30/the-coronavirus-pandemics-impact-on-pew-research-centers-global-polling/">global coronavirus pandemic</a> is a reminder that some factors cannot be accounted for by even the most thoughtful planning.</p><p>In Italy, we were able to conduct our survey via phone as part of the last wave of the Global Attitudes Survey. With the bridge survey in hand, we had a sense of possible measurement differences from switching modes and the impact on longstanding trends — and we now have a better sense of the issues we may find in other contexts for mode migrations.</p><p><em>Martha McRoy is a research methodologist focusing on international survey research methods at Pew Research Center.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4d04c3e9b8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pew-research-center-decoded/the-impact-of-survey-mode-on-measuring-public-opinion-in-italy-d4d04c3e9b8c">The impact of survey mode on measuring public opinion in Italy</a> was originally published in <a href="https://medium.com/pew-research-center-decoded">Pew Research Center: Decoded</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small changes in survey scales can matter when measuring political ideology in Europe]]></title>
            <link>https://medium.com/pew-research-center-decoded/small-changes-in-survey-scales-can-matter-when-measuring-political-ideology-in-europe-4a10d9a015c5?source=rss-847fef297fb5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a10d9a015c5</guid>
            <category><![CDATA[polling]]></category>
            <category><![CDATA[research-methods]]></category>
            <category><![CDATA[survey-research]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[political-science]]></category>
            <dc:creator><![CDATA[Martha McRoy]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 16:59:27 GMT</pubDate>
            <atom:updated>2020-09-01T19:35:10.499Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iCWVyDda-3eLcb5JChWt3g.png" /><figcaption>(Pew Research Center illustration/<a href="https://github.com/propublica/weepeople">Wee People</a>)</figcaption></figure><p><em>(Related post: </em><a href="https://medium.com/pew-research-center-decoded/how-much-do-the-numbers-used-in-survey-scales-really-matter-227d84ab2a13"><em>On a scale from 1 to 10, how much do the numbers used in survey scales really matter?</em></a><em>)</em></p><p>At Pew Research Center, survey questions about respondents’ political ideology are among the most important measures in our <a href="https://www.pewresearch.org/global/">comparative, cross-national surveys</a>. Our recent research in Europe, for instance, explored how <a href="https://www.pewresearch.org/global/2018/07/12/in-western-europe-populist-parties-tap-anti-establishment-frustration-but-have-little-appeal-across-ideological-divide/">political polarization</a> in the region shapes long-standing debates about domestic, social and economic policies.</p><p>To measure political ideology, our international questionnaires typically ask respondents to place themselves on a 7-point scale with two labeled endpoints (or anchors) — “extreme left” and “extreme right” — along with numbers for all of the points in between. However, <a href="https://pprg.stanford.edu/wp-content/uploads/1991-Response-strategies-for-coping-with-the-cognitive-demands-of-attitude-measures-in-surveys.pdf">some research</a> suggests that it’s easier for respondents to understand a fully labeled scale, one that includes descriptions for the interior points as well, such as “leaning left,” “leaning right,” or “center.” In this post, we’ll describe a survey experiment we conducted to find out if two approaches to measuring the same concept align with — or diverge from — each other.</p><h3>Findings at a glance</h3><p>Our survey experiment was part of nationally representative telephone surveys we fielded in France, Spain and the United Kingdom in the spring of 2018. (You can read about an earlier experiment related to survey scales <a href="https://medium.com/pew-research-center-decoded/how-much-do-the-numbers-used-in-survey-scales-really-matter-227d84ab2a13">in this Decoded post</a>.)</p><p>The 2018 surveys asked about political ideology in two ways. First, midway through each interview, respondents were asked the Center’s standard ideology question using a 7-point numeric response scale with only the endpoints labeled. Then, after several more questions, respondents were asked to describe themselves using a fully labeled, 5-point scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*JAGxdVgyG5eaA8Q_dfCrsw.png" /></figure><p>Here are some of the trends we observed across all three countries:</p><p>· The most common response option for both scales was the midpoint, though a greater share of respondents chose the midpoint on the 7-point scale (“3”) than on the 5-point scale (“center”).</p><p>· More respondents identified as right of center on the 7-point scale but left of center on the 5-point scale.</p><p>· Respondents were more likely to choose the endpoints on the 5-point scale than on the 7-point scale, and this was especially true for those on the ideological left.</p><p>· Levels of item nonresponse (either “don’t know” or refusal to answer) were similar using both approaches.</p><h3>A closer look at France using Sankey diagrams</h3><p>These aggregate numbers don’t tell the whole story. Fortunately, we can go a step further with our experimental data. By asking all respondents to answer both versions of our ideology question, we can also see how each respondent answered — or “moved” — from one question to the other.</p><p>To illustrate these patterns, we created Sankey diagrams for each country using the R package <a href="https://cran.r-project.org/web/packages/ggalluvial/vignettes/ggalluvial.html"><em>ggalluvial</em></a>. These graphs are a type of flow diagram that illustrate the movement from one category to the next — in this case, from the 7-point scale to the 5-point version. The band thickness in Sankey diagrams indicates the number of people taking a particular path, with thicker bands showing the more common pathways between two points on our different scales. (We should note at the outset that this part of our analysis does not include respondents who answered “don’t know” or refused to respond. It is limited to the 2,625 respondents who answered both ideology questions, including 846 in France, 935 in Spain and 844 in the UK.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*Ooqr4FFH7n2wqu2zHSlxCw.png" /></figure><p>In the graph on the left, we’ll focus on French respondents who answered “0 — Extreme left” on the ideology question using the 7-point scale. Of the 55 French respondents who reported “0 — Extreme left,” most of them (31) chose “left” when later asked to identify their political ideology using the 5-point scale. These respondents are represented by the highlighted horizontal line straight across the top of the chart.</p><p>But what about the 24 other people in France who chose the “0 — Extreme left” option? These respondents are represented by three narrower bands that start at “0 — Extreme left” and then slide down and to the right in the chart. Overall, most of these respondents (18) chose “leaning left.” Far fewer moved to “center” (five respondents) or “right” (one respondent). None of the respondents who answered “0 — Extreme left” on the 7-point scale reported “leaning right” on the 5-point scale.</p><p>The full range of patterns between the two scales in France is provided in the next Sankey diagram, with the path of each combination of responses illustrated by a different color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*_yJ4Ow-H5SFdfnX1i3igsA.png" /></figure><p>A few trends emerge in this graph:</p><p>· About half of the respondents (53%) who selected the midpoint on the 7-point scale also chose “center” on the 5-point scale, as illustrated by the thickest yellow band connecting “3” to “center.” Among those who chose the midpoint on the 7-point scale but did <em>not </em>choose “center” on the 5-point scale, roughly twice as many placed themselves left of center than right of center: 31% versus 17%.</p><p>· Nine-in-ten (90%) of those on the left of the 7-point scale identified as left of center on the 5-point scale as well, as illustrated by nearly all of the blue bands above the midpoint moving to either “left” or “leaning left.” Overall, 7% of those who characterized their ideology as left of center on the 7-point scale moved to “center” on the 5-point scale, while 3% moved to “leaning right” or “right.”</p><p>· Around two-thirds of respondents (66%) who placed themselves on the right of the 7-point scale also chose “right” or “leaning right” on the 5-point scale. Surprisingly, more of those who identified as right of center on the 7-point scale identified as “left” or “leaning left” on the 5-point scale (21%) than moved to the center (13%). This is shown by the similarly sized orange and red paths heading to “left,” “leaning left,” and “center.”</p><p>· Of the 35 possible response combinations between questions, no respondents in France answered “0 — Extreme left” on the 7-point scale and later identified as “leaning right” on the 5-point scale, or answered “1” and then identified as “center” or “leaning right.”</p><h3>Cross-national results from the experiment</h3><p>Sankey diagrams can also be helpful to illustrate and better understand cross-national patterns in our three selected countries. As the next chart shows, several of the broad trends we observed in France are apparent in Spain and the UK as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CHwuozU1brbiOOEmMqqbOA.png" /></figure><p>Here’s a closer look at what’s happening in this chart:</p><p>· Across all three countries, most respondents who chose the midpoint on the 7-point scale also chose “center” on the 5-point scale (53% in France, 52% in Spain and 59% in the UK). And just as in France, more people moved from the midpoint of the 7-point scale to <em>left </em>of center on the 5-point scale than moved to the <em>right</em> on the 5-point scale.</p><p>· In all three countries, the overwhelming majority of those who placed themselves on the left of the 7-point scale also identified as left of center on the 5-point scale (90% in France, 94% in Spain and 89% in the UK).</p><p>· There is more variability among those who placed themselves on the right of the 7-point scale. While 66% of those on the ideological right in France also placed themselves on the right of the 5-point scale, the share who did so was lower in both Spain (52%) and the UK (61%). Respondents on the ideological right of the 7-point scale in Spain and the UK were more likely than their counterparts in France to move to the <em>center</em> on the 5-point scale (26% in Spain and 25% in the UK versus 13% in France).</p><p>· As was the case in France, a few combinations of responses were not found in Spain and the UK. No respondents in Spain answered “0 — Extreme left” and then “center” or “leaning right,” or, in another combination, “1” and then a right-of-center response. Similarly, in the UK, no respondents first answered “0 — Extreme left” or “1” and then “right.”</p><p>As illustrated in the Sankey diagrams, there is general ideological consistency at the individual respondent level between the two survey questions, with only a limited share of people in each country providing apparently contradictory answers. In fact, only 11 of our 2,625 respondents provided the opposite extremes to the two scales (“0-Extreme left” and <em>right</em> or “6-Extreme right” and <em>left</em>). The robustness of these measures across countries suggests that both items can be useful in estimating political ideology. Across countries, our respondents behaved similarly, but not exactly the same. This is reflected in the subtle variability in responses to these two questions that are seemingly measuring the same thing.</p><h3>Concluding thoughts</h3><p>Both scales have benefits. A 5-point, fully labeled scale allows respondents to identify with a qualitative description of their political leaning, while a 7-point, endpoint labeled scale provides respondents with more choices to select a position closest to their political ideology. While the two survey questions are different in scale size and wording, respondents across countries answered them in similar patterns, giving us confidence in both when conducting cross-country comparisons.</p><p>There are several caveats, however. The experiment relied on relatively small samples that limited our findings in a few ways. For example, we could not conduct robust analyses across all possible answer combinations because some were too infrequent and others nonexistent. And while some respondents answered with “don’t know” or refused to answer at all, the number of people who did so was relatively small, limiting our ability to draw inferences about this group.</p><p>Aside from sample size, there are a few other issues we would consider expanding upon in future experiments. For instance, we did not rotate question order to examine whether replying to the first ideology question affected respondents’ answers to the second question or subsequent items. Also, we tested only two response scales, which means we could not assess the merits of other permutations, such as the results of a fully labeled, 7-point scale or a numeric, endpoint-labeled 5-point scale.</p><p>Finally, the questions we tested weren’t equivalent. The Center’s standard political ideology question includes the words “extreme” (in the 7-point scale itself) and “center” (in the question stem), but this is not the case in the alternative 5-point scale and associated question. Additional experimental conditions — say, keeping the scale length the same but testing different wordings, or vice versa — would allow us to disentangle these effects.</p><p>To dig deeper, it might be useful to look beyond the survey results. It’s possible some respondents were confused when we asked for their political ideology twice, so one way to explore the issue would be to incorporate <a href="https://www.hkr.se/contentassets/9ed7b1b3997e4bf4baa8d4eceed5cd87/gordonwillis.pdf">cognitive interviewing</a> in future experiments. This may help researchers better understand the meaning of survey questions, including questions that ask about the same underlying concept, from a respondent’s point of view.</p><p>In the meantime, we plan to use the data from this experiment to further study whether one of these ideology questions is a more accurate or valid measure than the other. Comparing each question with other items asked in our surveys — such as respondents’ political party affiliation or their opinions on highly politicized topics — would provide additional insights, with the help of Sankey diagrams to visualize the results.</p><p><em>This post was written by Martha McRoy and Stacy Pancratz. Both are research methodologists focusing on international survey research methods at Pew Research Center.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a10d9a015c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pew-research-center-decoded/small-changes-in-survey-scales-can-matter-when-measuring-political-ideology-in-europe-4a10d9a015c5">Small changes in survey scales can matter when measuring political ideology in Europe</a> was originally published in <a href="https://medium.com/pew-research-center-decoded">Pew Research Center: Decoded</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>