QUESTIONNAIRE

DESIGN
&
EXPERIMENTATION
BY
AMIT KUMAR
RAGUNANTHAN
 Face to face interview

•• Advantages
Disadvantages
– Direct interaction
–
– High
Claritycost
and display of exhibits
–
– Longer
Better timeand quantity of data
quality
– Higher response rate
–
–
Interviewer bias
No sequence bias
–
– Anonymity not maintained
Identifying respondents
–
– Interviewer cheating
Unstructured

– Time bias exists

– Field control needed

2
 Person Administered Surveys II

2. Indirect, non- face-to-face Interview:

• The interviewer and the interviewee do not see but
talk direct to each other.
– Telephone Interview

3
 Telephone Interview
• Advantages • Disadvantages
– Faster Results – No exhibits
– Inexpensive – Long interview not
– Better geographical possible
coverage – Inability to make judgment
– Irresistibility – Answering machines and
– Reaching hard-to-reach caller identification device
people – Sampling problem
– Timing: early or late OK • Obsolete directory:
– Privacy and better control poor sampling frame
– Coincidental data:
immediate feedback.

4
 Self-administered surveys
Survey
Mail • Disadvantages
• Advantages – Mailing list problem
– Wide geographical coverage
– Unidentifiable respondent
– Providing thoughtful answers – Questionnaire exposure
– Ability to ask sensitive questions – Data limitation
– No interviewer bias – No interviewer assistance
– Inexpensive • no exhibits
– Better control – Assumed literacy
– Anonymity – Poor response rate
– Clarity – Longer time

Survey through Internet has similar advantages and problems

5
 A questionnaire
[also called research instrument]

• Data collection • Six important functions

instrument used – Converts research objectives
for gathering data; into specific questions
• A formalized – Standardizes the questions
schedule of an – Keeps respondents motivated to
complete the research
assembly of a
– Serve as a permanent record
carefully
– Speed-up the process of data
formulated analysis
questions; – Reliability and validity purposes

6
 Questionnaire Development Process

Steps in a Q uestionnaire D evelopm ent P rocess

D eterm ine D ecide D a ta

Survey C ollection Q uestion
O bjectives M ethods D evelopm ent

Q uestion
P re-design activities E valuatio n
by R esearcher
and by C lient
D esign
P retest the activities
P ost-design activities Q uestionna
ire

T abulate and G ath er D ata R evise,

A nalyze D ata U sing the F inalize,
and F inalize Q uestio n- and
R epo rt naire D uplicate

7
 Basic Question Formats
V arious Q uestion Form ats

Basic
Q uestion F orm ats

O pen-ended C losed-ended Scaled

Q uestions Q uestions Q uestions

Basic D ichotom ous Labeled

O pen-ended Q uestions Q uestions
Q uestions

Probing M ultiple-C hoice U nlabeled

Q uestions R esponses Q uestions

C larify ing
Q uestions Single-coded

M ultiple-coded

8
OPEN-ENDED QUESTIONS: Questions to which respondents give their responses
freely, according to their own will.

Basic Open Ended Questions.

Q. What do you particularly like about Lipton Tea?

______________________________________________________________________________

Q. Why are you unwilling to buy a cellular phone when it is available in the market?

________________________________________________________________________

Probing Questions

Q. What do you particularly like about Johnson & Johnson baby oil?
__________________________________________________________________________

PROBE: Anything else?

9
 What should you ask?
• The questions asked are a function of previous
decisions
• The questions asked are a function of future
decisions (such as statistical analysis)
 Key criteria
• Questionnaire relevancy
– No unnecessary information is collected and only
information needed to solve the problem is
obtained. Be specific about your data needs; tie
each question to an objective
• Questionnaire accuracy
– Information is both reliable and valid
 Phrasing Questions
• Open ended response versus fixed alternative
questions
“?”
• Decision criteria: type of research; time;
method of delivery; budget; concerns
regarding researcher bias
 Avoid
• Leading questions
• Overly complex questions
• Use of jargon
• Loaded questions (can use a counterbiasing
statement)
• Ambiguity
• Double barreled questions
• Making assumptions
 Order?
• Order bias results from an alternative
answer’s position in a set of answers or from
the sequencing of questions
– Funneling technique: general to specific helps
understand the frame of reference first
• Anchoring effect: the first concept measured
tends to become a comparison point from
which subsequent evaluations are made
 Decisions
• Ranking, sorting, rating or choice?
• How many categories or response positions?
• Balanced or unbalanced?
• Forced choice or nonforced choice?
• Single measure or index?
 Advantages and disadvantages of
Open-ended Questions

Advantages Disadvantages
· Since they do not restrict the · Inappropriate for self-administered
respondent’s response, the questionnaire since people tend to
write more briefly than they speak.
widest scope of response can be
· The interviewer may only record a
attained. summary of the responses given by
· Most appropriate where the an interview and fail to capture the
range of possible responses is the interviewer’s own ideas.
broad, or cannot be · It is difficult to categorize and
predetermined. summarize the diverse responses of
different respondents.
· Less subject to interviewer bias.
· May annoy a respondent and prompt
• Responses may often be used as him/her to terminate the interview,
direct quotes to bring realism or ignore the mail questionnaire.
and life to the written report.

16
CLOSED-ENDED QUESTIONS: Questions to which respondents
are required to answer from set of alternative responses provided
by the researcher. Could be dichotomous or multiple choice.
Dichotomous Questions With No Neutral Response
_________________________________________________
Q.Do you have a cellular phone?
Yes ...................................... 1 without neutral
No ...................................... 2 response

Dichotomous Questions With Neutral Response

Q.Is it likely that you will purchase a cellular phone in the next
six months?
Yes ...................................... 1 with neutral
No ...................................... 2 response
Not Sure .............................. 3
17
 Single- and multi-coded multiple choice questions
_____________________________________________________________________
Q. On an average, how much do you spend on newspapers, books and magazines in a
month? (Please check one from the following responses.)

Less than $15 ................................... 1 Single-coded

Between $16 & $30 ......................…. 2 question
Between $31 & $45 .......................… 3
Between $46 & $60 .....................….. 4
$60 or more .................................….. 5

Q. Which of the following household appliances does your household have?

(Please check as many responses that are applicable to you.)

TV …………1 VCR …… … 5
LCD ………….2 Microwave ….…… 6 Multi-coded
PC …………3 Cellular phone ……….7 question
Fax …………4 Others ……….8
Specify ____________
__________________________________________________________________________________________________
__
18
 Advantages and Disadvantages of
Closed-ended Questions

ADVANTAGES DISADVANTAGES
· All respondents reply on a · Preparing the list of
standard response set. This responses is time-
ensures comparability of consuming.
responses, facilitates coding,
tabulating and interpreting the
· If the list of responses is
data. long, the respondents may
· Easier to administer and most be confused.
suited for self-administered · If the list of responses is
questionnaire. not comprehensive,
· If used in interviews, less skilled responses may often fail to
interviewer may be engaged to represent the respondent’s
do the job. point of views.

19
 Scaling questions

 Scaling questions are special types of closed-ended questions.

 They include, among others, the following categories of questions.
 Behavioral/Attitudinal questions
 Buying-intent questions
 Agree-Disagree questions
 Preference questions
 Ranking questions
 Semantic differential questions
 Constant-sum questions
 The questions can be labeled or unlabeled

20
 Labeled and Unlabeled Scaling Questions
Type of Examples Advantages Disadvantages
Scaled
Questions
Unlabeled “On a scale of 1 to 7, 1. Allows a respondent to Respondents
scaled- how would you rate express the degree of may not relate to
response the IBM Thinkpad on his/her intensity of the scale well.
question ease of operation? feelings.
2. Easy to administer and
code.
Labeled “Do you disagree 1. Allows a respondent to Scale may be
scaled- strongly, disagree, express the degree of “forced” or
response agree, or agree strongly his/her intensity of overly detailed.
question with the statement, feelings.
‘IBM laptops are a 2. Easy to administer and
better value than code.
Compaq laptops’?” 3. Respondents can relate
to the scale.
21
 Considerations in choosing a question format

• Nature of the property being measured

– Subjective Vs objective
• Previous research studies
– Need for comparison with past studies
• Data Collection Mode
– Telephone/face-to-face-interview/mail
• Scale level desired
– Statistical analysis
• Ability of the respondents

22
 Phrasing and Sequencing of Questions

PHRASING SEQUENCING
• Focus on a single issue or • Start with simple opening
topic questions
• Ask precise questions • Place
using respondent’s core – broad-based questions first;
– more specific and narrow
vocabulary
questions and difficult,
• Avoid sensitive, embarrassing
– use of vague words questions should come
– asking leading or loaded later(Funnel approach)
questions – Classification questions last.
– estimation questions • Transition from one topic to
– double barreled questions another should be smooth
– presumptuous questions

23
 Questionnaire Layout

• Provide sufficient spaces

• Use prominent print for instructions
• Use filtered questions
• Do not slit the same question over two
pages
• Number the questions
• Layout should facilitate editing and coding

24
 Comparative Evaluation of Various survey Methods.
________________________________________________________________________
Face-to-face Interview
Criteria In-home/ Mall- Telephone Mail
In-office intercept Interview Survey
Flexibility of data collection High High Moderate Low
Diversity of questions High High Low Moderate
Sample control Potentially Moderate Moderate to Low
high high
Control of data collection Moderate to High Moderate Low
environment high
Response rate High High Moderate Low
Show of exhibits Yes Yes No No
Quantity of data High Moderate Low High
Anonymity of the respondents Absent Absent Moderate High
Access to sensitive information Low Low High High
Speed Moderate Fast Fast Slow
Cost High Moderate Moderate Low
Potential interviewer bias High Moderate to Moderate Low

25
 Design of Experiments

• Goals
• Terminology
• Full factorial designs
– m-factor ANOVA
• Fractional factorial designs
• Multi-factorial designs

Copyright 2004 David J. Lilja 26

 General Procedure
• Determine upper/lower bounds for parameters
• Simulate configurations to find response
• Compute effects of each parameter for each
configuration
• Rank the parameters for each benchmark based on
effects
• Average the ranks across benchmarks
• Focus on top-ranked parameters for subsequent analysis

Copyright 2004 David J. Lilja 27