The Long and the Short of Questionnaire Length

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Over and over again I'm hearing researchers complain about questionnaire length.  At the AMA MR conference a few weeks ago, at the Gartner CRM conference in early September and at the MRA conference this June, I heard the same complaints:

• "We're making our questionnaires so long that respondents are dropping off half way through the survey."
• "When we have longer questionnaires, respondents become fatigued; they don't pay as close attention to their answers.  They rush to the finish; the quality of the results is compromised."
• "When we survey customers, they complain if the questionnaire is too long."

In fact, in an audience poll at an AMA session, 26% of attendees said researchers need to limit the number of questions they ask, in order to increase online response rates.

So, it's a common refrain, but how do we actually refrain from asking too many questions?  The advice tends to be:  "Don't ask so many questions!"  That's hard advice to follow.  Questionnaires, like Congressional bills, have a natural tendency to grow longer.  Everybody has questions that they want to include.

So here are some practical tips for shortening questionnaires:

• Answer the Questions Yourself - For customer and prospect surveys, take a hard look at the questionnaire to see which questions can be answered by looking up the data from the appropriate systems.  If this is a questionnaire following up to a customer service call, for instance, since your call-center software tracks how long the call was and stores a code indicating the type of inquiry, you don't need to ask the respondent for that information.  For that demographic section at the end, your CRM system should have many basic facts about the respondent, such as their age, gender and address.   Taking the time to integrate your community feedback system with your other systems can easily cut 10% of the questions you ask, and those are often the most tedious and annoying questions to answer, from a respondent perspective.
• Don't Ask All the Questions - Use skip patterns and randomization to show respondents different subsets of the questionnaire.  At the IMC conference, Coast Plaza Hotel staff handed out short paper surveys to luncheon attendees.  A few of us discussed the questionnaire, and-to my surprise-while some key questions were the same, the list of attributes to be rated varied from survey to survey.  The hotel was able to gather detailed information across many aspects of its services without overwhelming any individual respondent.
• Ask Only the Most Important Questions - A common research tactic is to address a key issue with three or more questions, all with very similar wording.  This is important for benchmarks;  for instance, the Norwegian Customer Satisfaction Barometer has three questions around corporate image, asking the respondent to rate a corporation's image, asking them to think of how their friends' would rate the corporate image, then asking them to rerate the corporate image when thinking of how well the company does compared to competition.  If you're not benchmarking a particular measure, reduce it to one question.
• Don't Ask Questions the Respondent Wouldn't Understand - Sometimes surveys go into excruciating detail about very minor aspects of a product or service.  This is understandable, from the perspective of the corporation, which is responsible for every tiny detail.  However, many of these distinctions are too subtle for the respondents.  The questionnaire might use industry and technical terms that respondents don't know or misunderstand.   For cases like this, conduct some qualitative interviews by phone, focus group - or most efficiently - by using your online community;  this research can make sure that you are using language the respondent understands and making distinctions that are important to the respondent.  Where you are not, get rid of those questions.
• Don't Ask Questions for the Sake of Asking Questions - Look hard at the questionnaire to see if the answer to a question will help you with your current decision making.  Sometimes, when writing a questionnaire, you begin to think "It would be nice to know...".  How would knowing this particular answer help you?  If it wouldn't, then get rid of the question, or-if you really can't stomach that-show it only to a random subset of respondents.  For instance, I often seeing pricing questions inserted into surveys that aren't about pricing; these are good candidates to postpone to some later study.
• Survey More Frequently - Finally, if you have many questions on a different topic, then considering running a separate survey on that topic later.  The less frequently your organization conducts surveys, the greater the desire to squeeze everything into this survey.  By doing surveys more often, you alleviate this.  And by using random sampling and enterprise feedback management, you can make sure that you are not oversurveying.

So, how do you make questionnaires shorter?  The key secret is to shorten the questionnaire from the respondent's perspective.  Hiding questions, showing random subsets of questions and asking questions in a later survey are all ways to shorten this particular survey from an individual's perspective.

Rather than hearing continued complaints about questionnaire length, I hope at next year's conferences that I instead hear people comparing techniques for shortening questionnaires!