Writing Objective Survey Questions

Email Article |  Facebook |  Twitter |  LinkedIn 

The saying Garbage In, Gospel Out reflects our willingness to believe computer output, even if it was generated from bad input. Survey researchers are no more immune to this tendency than computer scientists, as poorly worded questions can lead to suspect results and erroneous conclusions.

We recognize bad questions when we see them in other people's surveys, but we don't recognize them as easily when we write them ourselves. Each of the following examples are adapted from public surveys:

1. "How likely is it that you will attend the 2009 Expo at our new, low entry prices?"
2. "The 2009 Forum will be held July 24-26 in central Iowa. Please check all the reasons why you may choose not to attend."
3. "What type of involvement would you like to have with the Celebration of Nations program?"
4. "What do you think of those titles? Yes, we are clearly not creative, but that's why we are asking for your help."

• Your questions should use nonjudgmental wording and neutral terms. Respondents should not be able to determine where you stand on any topic. (See this post for an example of how to research attitudes towards abortion.)
• Don't presuppose anything (one type of leading question). For the third example above, for instance, ask instead "What type of involvement, if any, would you like to have with the Celebration of Nations program?"
• Don't ask other types of leading questions. (See the links for details.)
• Avoid so-called "double barreled questions" by splitting them into two. Instead of "How would you rate our price and service?" ask "How would you rate our price?... How would you rate our service?" 
• Remove ambiguity in use of words and grammatical structures.  The question "Will a bimonthly schedule make you more or less likely to renew your subscription?" is useless, since bimonthly can mean either "twice a month" or "every other month".
• Avoid industry jargon and acronyms; too often in my own surveys I've assumed people know what I mean by CRM or TQM when I should have defined and described them.
• Specify how you use general terms. If you need to map results back to industry figures, make sure you are using industry definitions. Instead of "Have you purchased a new big-screen television in the past year?" make sure you define the cut-off screen size that maps to your data (e.g., "Since January 1, 2008, have you purchased a new big screen (40-inch or larger) television?").
• Open-ended questions should specify a unit of measure. Instead of "How far do you live from the nearest Acme store?" ask "How many minutes does it typically take you to travel to the Acme store that you typically shop at?"

In general, try to write from the respondent's perspective rather than your perspective. Don't make subtle distinctions that would be obvious to a coworker but not a customer. You spend 40, 50, 60 or more hours a week thinking about your company and its products and services; your customers don't. Have others outside the organization proofread your questions for clarity. For strategic surveys, pre-test your survey with a segment of your audience.

Bad questions are the Garbage In, with their analysis being the Garbage Out. A question like "How likely is it that you will attend the 2009 Expo at our new, low entry prices?" will end up overstating actual likelihood to attend, while a question like the following will understate the likelihood to attend: "The 2009 Forum will be held July 24-26 in central Iowa. Please check all the reasons why you may choose not to attend." 

If you don't get the question right, you won't get the analysis right.