Satisficing and Survey Respondent Behavior

As survey authors, we ask a lot of our respondents-literally and figuratively. In "Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys", Jon Krosnick outlines four mental steps that respondents work through to answer a survey question:

1. Interpret the meaning of a question
2. Recall all relevant facts related to the question
3. Internally summarize those facts
4. Report summary judgment accurately 

Ideally, we want responses to carefully think through every answer they provide, but: the respondent is not a machine, your survey is not a test. Not every respondent is as conscientious as we'd like.  They realize they are not being graded on their answers. This will not affect their permanent record.

As a result, respondents take shortcuts, which Krosnick describes as satisficing, an economic term that is a blend of satisfy and suffice. In economics, consumers are supposed to select the optimal product or service for their needs, but in reality consumers often choose a satisfactory solution rather than the optimal solution, because it is too time consuming to evaluate all the available alternatives. In surveys, respondents are supposed to select the optimal answer, but in reality respondents often choose a satisfactory answer instead.

When a respondent works through each of the four mental steps with a maximum level of effort, they are optimizing their answers.  When a respondent works through each step with minimal effort, they are engaging in weak satisficing, according to Krosnick. When they skip the steps of recall and internal summarization, they are strong satisficing.  When they are simply jumping to a conclusion or selecting any answer, I consider them to be cheating.

 

Some of the behaviors that are weak satisficing include:

• Selecting the first choice that appears reasonable in a list of choices, rather than reading the entire list
• Agreeing with assertions without thinking the contrary position through ("acquiescence response bias")

Strong-satisficing behaviors include:

• Endorsing the status quo instead of change
• Failing to differentiate in ratings
• Selecting "Don't know" rather than expressing an opinion
• Randomly choosing an answer (which Krosnick categorizes as strong satisficing but which I consider to be cheating)

A well-designed survey needs to account for these behaviors.  Some tips from a follow-on paper by Krosnick, et al, "Satisficing in surveys: Initial evidence":

• In the introduction, let the respondent know the importance of providing accurate answers.
• Inform respondents of the value of their input and how its results will be used.
• After  some questions, ask the respondent to justify their answer with an open-ended question.
• Re-factor complex questions into simpler questions (especially re-factor matrix questions, see SSI's just published study on Grid Questions).
• Rewrite questions for a lower reading level.

Some tips of my own, some specific to web surveys:

• When a choose-one question has no logical order, randomize the list of choices.  If you don't do this, a choice near the top of the list might be selected more often than it deserves.
• To avoid acquiescence, recast agreement questions as offering multiple alternatives rather than one implicit alternative (disagreement).  Some assessments later in the questionnaire ask for agreement with the inverse of an earlier statement to test respondent reliability; this lengthens the questionnaire and should be done sparingly.
• Avoid Yes/No questions that no one wants to say no to.
• On rating scales, omit a "Don't know" option to force respondents to select a rating.  (Continue to use "Don't know" options when asking respondents to recall past action and choose from a list of alternatives.)

Respondents aren't machines, executing questionnaires in their heads like source code, and questionnaire designers need to plan accordingly.