Multiple-Answer Questions (“Select All That Apply”): Best Practices

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

A common question type in most  survey software applications is the "Choose Many":

The best practices for using this question type are straightforward:

• Always include a "None of the above" as an exclusive choice - If you omit this option, then a valid response is to not select any of the choices, because none of them apply. Nothing is more maddening for a respondent then to purposefully not answer this question only to have it be required, with the survey system prompting for an answer. To handle this case, set the last option to "None of the above" or similar wording and set it to be exclusive: if the respondent clicks this choice, the other checkboxes will be automatically unchecked. If you do not require an answer, failure to use this option will give you an ambiguity in the data: did the respondent skip the question or "answer" it by not checking anything? (If your survey software doesn't include the ability to make this exclusive, include the choice but manually clean the data beforehand, removing any other choices selected when "None of the above" was selected.)
• If the choice list has no natural order, randomize the order - Respondents are in a hurry and have a tendency to skim the later choices of a long choice list (see Satisficing and Survey Respondent Behavior).  If they have checked a few choices early on, then they feel like they have answered this question sufficiently and will move on to the next.  By randomizing the list, you avoid giving an artificial bias to early options.  Of course, make sure to anchor the position of "None of the above" choice to the bottom of the list.  
• Avoid providing choices that overlap in meaning - If one choice overlaps another, then respondents will be inconsistent about selecting both choices when appropriate.
• Explicitly remind respondents to select all that apply - Include instructions such as "Please check all that apply" at the end of your question text [suggested by Kevin Schulman]
• Don't use a listbox to show the choices - While the HTML <SELECT> tag has long supported the ability to use this user-interface control (see illustration), it is confusing to respondents and rarely used today. Checkboxes are the better choice.
• Consider using a Yes/No matrix - As discussed in my recent post, "Yes/No Matrix Questions", sometimes a list of Yes/No questions is more appropriate to your research than a choose-many question.

Two powerful applications of the choose-many question:

• As a preface to a matrix question - Ask the respondent all the brands that they are familiar with, then - on the next page - show a matrix question asking them to provide additional information about each brand they selected. See "Questionnaire Workshop: Refactoring One Question into Three Parts" for an example of this approach. In addition to piping selected choices into a matrix, Vovici v4 survey software supports the ability to pipe unselected choices into a matrix.
• To select the most important items - When you have more choices than a respondent can comfortably rank (8 or more), instead ask the respondent to "choose at most 3 choices". While you can ask the respondent to "choose exactly 3 choices", and set the validation to require this, it is better to let them select up to 3 choices, rather than have to include a lower priority choice in the list.

Ironically, I picked the choose-many topic as the subject of this morning's blog post because I didn't think I had much to say about it! And now I feel like I am leaving something out. What other aspects of this question type are important to consider?