Surveyese Spoken Here... Unfortunately
Posted by Jeffrey Henning on Fri, Dec 17, 2010
Ask someone who hasn’t written a questionnaire before to write one, and they are going to strive to create something that sounds like a survey. They’ll come up with questions like:
- On a scale of 1 to 10, where 1 is poor and 10 is excellent, overall how satisfied are you with our service?
- Please rank each of the features below, where 1 is most important and 15 is least important.
- What percentage of your work week do you spend on each of the following tasks?
- For each of the following brands, are you a current user, former user, aware, or unaware?
Sounds like a proper survey, doesn’t it? It’s as if Surveyese is a dialect that they’ve heard all their life, and now they are trying to speak it like a native. People assume that these techniques, which look so formal and official, are the right way to do things.
They’re wrong. Writing good questionnaires is hard, and many practices they’ve seen in a lifetime of taking surveys are outdated and don’t reflect modern best practices.
- Numeric rating scales seem like standard Surveyese, but fully labeled rating scales provide more reliable results and are easier on the respondent. “Overall how satisfied are you with our service? Not at all satisfied, slightly satisfied, moderately satisfied, very satisfied or completely satisfied?” The answers to such questions are also more consistent from respondent to respondent.
- Ranking questions are also a staple of surveys, but they are mentally taxing on respondents: in the example above, 15 items are way too many items to have to compare and sort into order. Nor are ranking questions easy to analyze: the distance between someone’s first and second choice might be much wider than the difference between their second and seventh choices and, again, varies from respondent to respondent. Rewrite ranking questions as choice lists (“Which of the following features are most important to you?”) or, if you must, as rating grids. For more sophisticated needs, use discrete choice modeling.
- Allocation questions also seem like proper Surveyese. As I review some questionnaires, I get the sense that the survey author thinks a good survey should treat a human like a robot, turning their satisfaction into a number, assigning a numeric ranking to each feature, and allocating their time to each task like some sort of cyborg spreadsheet. Assuming that someone really knows the right answers to an allocation question places too much faith in the respondent.
- Grids may be the hallmark of Surveyese: nothing shouts “This is a Real Survey” better than a nice big grid with lots of columns. Avoid the temptation, and find ways to split up that matrix. For the brand question above, ask them first to choose the brands they are aware of from a multi-select list of checkboxes. Then ask them, showing just the brands they’ve heard of, to tell you their usage of each brand.
If bad money drives out good, then bad questionnaires drive out good. The more bad surveys people see, the more new survey authors think that these Surveyese techniques are the right way to proceed. It’s a never ending cycle of Do-It-Yourself deflation of survey quality.
Instead, we should all relentlessly seek to simply our questionnaires – using fewer words and asking fewer questions, all done in a conversational but objective fashion. Don’t ask the respondent to quantify everything in their life – ask them about themselves in plain language that they understand. Because, at the end of the day, respondents aren’t as fluent in Surveyese as we assume they are.
If we all start writing questionnaires in plain English, then one day Surveyese will be a dead language.
See also: