Welcome to the Listening Post!

Response Substitution: "Answering the Unasked Question"

Tweet

 

In the Journal of Marketing Research, David Gal and Derek D. Rucker, authors of the article "Answering the Unasked Question: Response Substitution in Consumer Surveys", suggest a newly observed respondent behavior that can reduce the quality of survey data. "We propose that respondents' answers to questions might sometimes reflect attitudes that respondents want to convey, but that the researcher has not asked about, a phenomenon we term response substitution."

Research magazine provided a nice summary of the phenomenon:

For example, a consumer completing a survey about a restaurant where they received great food but poor service would use the survey as an opportunity to express the dislike they felt toward the organisation, even if the survey focused on the quality of the meal.

Gal said: "Since the survey does not ask about the service at the restaurant, the only means for this unhappy customer to express his general disapproval of the service is by being negative about the food. Basically, respondents have the potential to project their negative attitudes onto the wrong aspect of the business so they can share their opinions. This bias can be a major source of misdirection for a company."

The authors wrote, "Our findings cannot be explained by the well-known halo-effect, namely, individuals' tendency to judge an object or person about which they hold a general negative attitude more negatively on other attributes... In contrast, we found that providing a negative evaluation of wastefulness prior to evaluating intelligence actually leads to a more positive evaluation of intelligence, the opposite of what a halo account would predict."

In comments on the Research article, Gary Halpern wrote, "I am not surprised to learn this. In fact, we believe that the occurrence of ‘venting' responses are as common, if not more common, than the occurrence of socially acceptable responses."

While the authors didn't discuss the impact on grid questions, it does make me wonder if longer grid questions would be less prone to response substitution, as respondents assume that one of the rows will cover the topic important to them. This research also raises a potential problem with surveys that are too short: respondents may be answering the "unasked question" instead of the posed questions.

To minimize response substitution, the study's authors suggest telling respondents that they will have an opportunity to express any other thoughts in a later open-ended question, before showing them the battery of closed-ended questions. As an alternative, I would suggest beginning the questionnaire or section of the questionnaire with open-ended questions instead; not only does this provide a better "escape valve" for this venting but it lets you capture respondent attitudes before you have biased them with the topics you have selected for closed-ended questions. Response substitution is a complex respondent behavior that may have a simple fix.