Yes/No Questions: Common Pitfalls to Avoid When Writing Questionnaires
Posted by Jeffrey Henning on Wed, Mar 04, 2009
Nothing could be simpler than asking a Yes/No question in a survey, right?
Sorry, that was rhetorical.
Here are some of the common mistakes I see made when surveys include Yes/No questions. All of the examples are based on actual questionnaires I've reviewed, with the wording paraphrased to protect the guilty.
Asking for a single Yes/No to multiple items.
Example: Was service prompt and courteous? Yes/No.
Someone who found the service prompt but not courteous (or vice versa) will probably feel conflicted but answer "No". When you write such a question, chances are that you are thinking of the two items as synonymous (as in "Was the policy clear and easy to understand?"); however, you've actually embedded two separate topics. Another example: "Do you think our campaign was effective at registering and mobilizing new voters?" A respondent might think the registration work was effective, but not the mobilization to the polls.
Solution: Split the problem question into two separate questions. "Was service prompt?" "Was service courteous?"
Letting the user select both Yes and No.
Example:
This may seem comical to you, but I've seen it literally dozens of times. Many first-time survey authors don't distinguish between checkboxes and radio buttons.
Solution: Make sure to use a "choose one" or "single select" question type for all of your Yes/No questions.
Force-fitting a question into a yes/no format by overriding what "Yes" or "No" means.
Example: Did you watch the documentary "No Direction Home: Bob Dylan"? (Please vote "yes" if you watched at least half of the film).
Solution: Either reword the question ("Did you watch any of the documentary...?") or instead give the user a list of choices to choose from: "How much of the documentary ‘No Direction Home: Bob Dylan' did you watch? (a) None of it (b) Less than half (c) About half of it (d) More than half (e) All of it."
Providing caveats to the Yes/No choices.
Example:
Have you been personally affected by asthma?
- No
- Yes, someone I care about has been affected by asthma
- Yes, I am a sufferer
The author started to write a Yes/No question, then decided that there were finer points that were important to capture. The problem is that these addendums to the choices are typically a sign that author did not think the question through thoroughly. For instance, how would someone who once had asthma but no longer does answer this question? (Never mind the poor word choice of a subjective term like "sufferer"; many respondents would see that term as too negative.)
Solution: Rewrite the question as a close-ended question with a thorough list of choices or as a series of Yes/No questions. "Has someone you care about been affected by asthma? Yes/No. Have you ever been diagnosed with asthma? Yes/No."
Asking questions that no one wants to say "no" to.
Examples:
Did you think using Facebook to reach young voters politically was an effective strategy?
Do you think we should continue to organize creative political initiatives?
Solution: Instead ask the respondent to rate the items on a scale. "Please rate the importance of using Facebook to reach young voters on a scale of 0 to 10, where 0 is not at all important and 10 is very important."
Asking questions that can't be answered Yes/No.
Example: Should customer-service reps place a caller on hold while researching an answer, or is it okay to ask to call back? Yes/No.
Hard to believe this was even posed as a Yes/No question, but this is a paraphrase of just such a question. A survey author may write something like this after writing a series of Yes/No questions in a row.
Solution: Rephrase as a question with the appropriate choices. "When faced with a time-consuming issue to research, should the customer-service representative (a) place the caller on hold or (b) ask to call back once the answer is found?"
Listing a bunch of similar yes/no questions in a matrix.
Example:
Did the ombudsman greet you promptly?
Did the ombudsman explain the appeals process clearly?
Did the ombudsman answer all your questions about the appeals process?
Did the ombudsman treat you with respect?
Did the ombudsman call you after you received the decision?
Solution: Rephrase to use checkboxes instead. This will be much more concise.
Did the ombudsman...? (Please check all that apply.)
Greet you promptly
Explain the appeals process clearly
Answer all your questions about the appeals process
Treat you with respect
Call you after you received the decision
None of the above
Too often, Yes/No questions are a cop-out for the busy or overwhelmed survey author. Often, you really need to give the respondent more choices. When that's the case, write a clear list of choices instead; perhaps review the best practices for writing closed-ended questions. Nothing could be simpler, right?