Yes/No Matrix Questions
Posted by Jeffrey Henning on Fri, Mar 13, 2009
As I reviewed questionnaires with Yes/No questions to identify common mistakes, I encountered quite a few Yes/No matrixes. Here's one of my own:
An interesting decision for the survey author is whether or not to show such questions as choose-many questions (checkbox lists) instead:
Some researchers have suggested that converting select-all-that-apply questions to Yes/No matrix questions ("forced-choice questions") with each answer required will improve the quality of results. The rationale is that respondents will need to spend more time answering each question.
Certainly, this question format is more time-consuming, since respondents must click a choice for every single item, whereas with a choose-many question they only need to check the items that apply. Since I am usually in favor of reducing the burden on the respondent as much as possible, I would normally never even consider using Yes/No matrices instead of checkboxes.
But before I rule in favor of the choose-many question in this case, let's hear from the defense. In the paper "Multiple Answer Questions in Self-Administered Surveys: The Use of Check-All-That-Apply and Forced-Choice Question Formats" [PDF] - by DA Dillman, JD Smyth, LM Christian and MJ Stern - the authors found that respondents select more choices in the force-choice format (Yes/No or equivalent labels) than in the standard checkbox format. Which format is more accurate is yet to be determined. The authors write that "forced-choice formats cannot simply replace check-all-that-apply formats and decisions about which format to use should be evaluated for each individual survey and question."
Such a lukewarm recommendation is the burden of a scientific researcher. As a blogger, I bear no such burden and can happily pontificate about when you should use which format. By default, I would use the checkbox format. The only exception I would make is when the list of possible choices is long (more than 7 items) and it is imperative that no choice item that should be selected is skipped. The Yes/No matrix is a speed bump in the questionnaire, and the respondent will pay more attention to each of its items than in a checkbox list.
Here are two of the few applications I can think of for the Yes/No matrix:
- When, as part of a technical support satisfaction survey, you are asking respondent to report all the steps they took as part of the problem resolution, consider using the Yes/No question matrix to make sure that no step was missed.
- When asking a consumer immediately after a retail purchase to recall a list of specific actions taken by the cantina staff, since some retailers grade their outlets based on franchise consistency.
And for those of you who think I should have included Clone Wars in my list of Star Wars movies, I confess that I gave up on George Lucas while he was still making live-action Star Wars films. I haven't even seen all three prequels. Sad words to write for a kid who had the entire set of Burger Chef Star Wars movie posters in his room.