Welcome to the Listening Post!

Zen & the Art of Questionnaire Design

Tweet

 

In A Fish Called Wanda, Jamie Lee Curtis says to Kevin Kline, "Now let me correct you on a couple of things, OK? Aristotle was not Belgian. The central message of Buddhism is not ‘Every man for himself.' And the London Underground is not a political movement."

Despite reading Zen & the Art of Motorcycle Maintenance, Zen in the Art of Archery and even The Tao of Pooh, I don't know much more about Buddhism than Kevin Kline's character did. But I do know a bit about questionnaires, so here is my Eightfold Path to Survey Enlightenment:

1. Right Perspective--The Respondent as Human. When writing a questionnaire, we tend to forget that we are asking a lot of the respondent, literally and figuratively. Answering survey questions takes at least four mental steps, and respondents sometimes skimp or even skip those steps, especially the further into the survey they are. Remember that respondents are human, not machines: keep the questions short and simple, and take into account natural behavior such as satisficing and acquiescence. (See Satisficing & Survey Respondent Behavior.)
2. Right Intention--Worthy Purpose for Survey. In the case of archery, the hitter and the hit are no longer two opposing objects, but one reality. How can you make your questionnaire one in reality with your research goals? Don't jump into a survey until you've articulated the Essential Question the research must answer, and then make sure that a survey is the best approach. (See Good Surveys start with Good Goals.)
3. Right Order--Questions' Proper Sequence. Every day life is the path. At the AAPOR annual conference last week, Kristin Stettler of the U.S. Census Bureau discussed having an interview funnel questions from the general to the specific and then the very targeted. This pattern works well for questionnaire design. While observing that structure, strive to keep the flow conversational. (See Order Questions Logically.)
4. Right Attention--How Each Question Listens. Every question type represents a trade-off: open-ended questions offer rich insights, but are more exhausting for respondents than closed-ended questions; ranking questions force respondents to differentiate items, but are harder to answer than rating questions; matrix questions can be answered quickly, but provide lower quality answers than separate questions. Weigh your options carefully. (See Types of Questions: Four Building Blocks for Constructing Questionnaires.)
5. Right Speech--Asking Objective Questions. Nan-in served tea. He poured his visitor's cup full, and then kept on pouring. "Like this cup," Nan-in said, "you are full of your own opinions and speculations. How can I show you Zen unless you first empty your cup?" Respondents should not be able to determine from your question wording where you stand on any topic. Use nonjudgmental wording and choose neutral terms. Empty your cup! (See Writing Objective Survey Questions.)
6. Right Choice--Prompting with Care. Questions with short choice lists are more likely to misrepresent what the target population really feels than questions with longer choice lists. While researchers and organizations have their favorite styles of rating questions, fully-labeled five-point rating scales produce results with the highest reliability and validity. Set aside yourself and follow the enlightened one. (See Rating Scale Best Practices.)
7. Right Length--Not Too Short, Not Too Long. Ideally, you should keep most surveys under ten minutes long. Employees, partners and major customers will tolerate longer questionnaires, and may in fact be offended by questionnaires that are too short. Strike the right balance. (See Recommended Survey Length.)
8. Right Action--Fulfilling the Survey's Karma. Two hands clap and there is a sound. What is the sound of one hand? You ask your respondents for information, but you give them no information back. Let them know, at a high level, what you learnt. Let them know how you will change and improve. Otherwise, they will be less likely to take your next survey. Survey karma's gonna get you. (See Closing the Feedback Loop: Sharing Results with Online Community Members & Respondents.)

Get a questionnaire wrong, and the data that you get back will be wrong. Follow the Eightfold Path to get your questionnaire Right.

Check out our popular webinar of the same title, Zen & the Art of Questionnaire Design. If you've seen it before, you can listen to an extended question and answer session from our most recent webcast.