Welcome to the Listening Post!

Understand Respondent Behavior to Write Better Surveys

Tweet

 

In the ideal world, respondents would carefully read every question, would thoughtfully contemplate an answer and would select the most appropriate response to each question. In the real world, many factors muddle responses.

Good questionnaire design has to start with understanding the ways respondents behave when taking surveys.

Cognitive Behaviors

• Satisficing is when respondents take mental shortcuts when answering questions, rather than carefully thinking through each question.
• Memory biases include the inability to recall precise quantities or frequencies, sensory details after the fact or how recently or far in the past something happened. (Wikipedia lists over 40 types of memory bias.)

Social Behaviors

• Acquiescence bias is the tendency of respondents to agree with statements made by the survey author rather than to give their honest opinion.
• Social desirability bias is the tendency of respondents to provide the politically correct answer rather than the candid answer. This stems from an innate desire to be seen favorably by others.
• Economic behavior, a frequent subject of survey research, is not strictly rational but is susceptible to framing, emotional arousal and heuristic decision-making.

Survey Behaviors

• Response styles are habitual patterns for answering survey questions: some respondents only use endpoints of a scale (Extreme Response Style), while some will only use the middle of the scale (Midpoint Response). These personal preferences often cause response styles to vary by country, making it difficult to compare the results of multinational studies.
• Response substitution is when respondents’ answers don’t address the topic raised by the questionnaire but instead reflect attitudes they want to convey about other topics. For instance, if the food was bad at a restaurant but all the questions were about its service, the respondent would rate the service worse than it was.
• Halo error is when general satisfaction causes respondents to inflate their ratings of different aspects (halo effect) or when dissatisfaction causes respondents to lower their ratings (reverse halo effect).
• Mode effects reflect the fact that respondents do not answer questions the same way in person, on the phone, on paper or via the web.
• Practice effects and panel conditioning impact the behavior of respondents in subsequent surveys. The more often respondents take surveys, the more accurate their answers are. Of course, the more surveys they see from your organization the less like the general population they may be in their knowledge of your organization.

Please see each article linked to above for practical tips on how to design surveys that address these respondent behaviors.