Order Effects: Early Choices in Long Choice Lists Are Selected More Often than Later Choices

We last looked at respondent behavior with the post Long Surveys Turn Respondents into Liars. Well, similarly, long choice lists turn respondents into satisficers, selecting a satisfactory answer rather than the optimal answer.

Jon Krosnick and Duane Alwin in the report "An evaluation of a cognitive theory of response order effects in survey measurement" provide an excellent summary of the past research that documented this behavior:

Studies of impression formation¹, the impact of persuasive communications², sequential processing of performance information³, and the serial position effect⁴ all suggest that when items are presented visually on "show cards," primacy effects are to be expected. This occurs for two main reasons. 

1. Items presented early may establish a cognitive framework or standard of comparison that guides interpretation of later items. Because of their role in establishing the framework, early items may be accorded special significance in subsequent judgments.
2. Items presented early in a list are likely to be subjected to deeper cognitive processing; by the time a respondent considers the final alternative, his or her mind is likely to be cluttered with thoughts about previous alternatives that inhibit extensive consideration of it. Research on problem-solving suggests that the deeper processing accorded to early items is likely to be dominated by generation of cognitions that justify selection of these early items⁵. Later items are less likely to stimulate generation of such justifications (because they are less carefully considered) and may therefore be selected less frequently.

So, now that we know that our respondents do this, how do we address this issue when constructing choice lists?

• If a long choice list can be structured into an outline, present the choices as a hierarchical question instead.
• Consolidate the long choice list into a shorter list that makes fewer distinctions.
• For long lists that can't be modified, use randomization. While it would be too costly in a paper survey to have multiple versions of the questionnaire, each presenting choice lists in different orders, for a web survey the ability to randomize choice lists is a built-in capability of most survey software and has no added cost to use. Such randomization isn't needed for long lists that respondents don't have to read; for instance, alphabetized lists of states or countries, where the respondent knows the answer without reading the choice list and is simply finding the choice in the list. Nor is randomization appropriate for rating scales. Instead, randomize the choices for any long list that lacks an inherent order.
• Finally, Krosnick and Alwin advise attempting to "to increase respondent motivation in order to increase concentration and decrease satisficing. Motivation may be increased by adding special instructions informing respondents that the question they are about to answer is relatively difficult and requires extra concentration." 

¹ Asch, 1946;Nisbett & Ross, 1980, p. 172-175; Anderson & Hubert, 1963; Sherif, 1935; 1936; Lingle & Ostrom, 1981; Anderson L Barrios, 1961; Dreben, Fiske, & Hastie,1979. ² Miller & Campbell, 1959; Ronis et al., 1977; Crano, 1977; Hovland et al., 1957; Insko, 1964. ³ Jones et al., 1968. ⁴ Bruce & Papay, 1970; Crowder, 1969; Rundus, 1971. ⁵ Koriat, Lichtenstein, & Fischhoff,1980; Hoch, 1984; Klayman & Ha, 1984; Tschirgi, 1980; Wason & Johnson-Laird,1972.