Survey Test Mode
Posted by Jeffrey Henning on Thu, Jun 25, 2009
I hate that every gas station near me has disabled the ability for me to set the fuel dispenser to fill the tank and then walk away, safe in the knowledge the automatic cut-off will kick in. Now I have to stand at the pump manually holding the switch on the fuel dispenser until the tank is full. No doubt this is a safety innovation in case the auto cut-off failed.
In a similar way, many users of our survey software dislike the fact that the first time they publish a new survey, it goes into test mode. This is a not-so-subtle reminder that they should double-check the survey before inviting participants to respond.
Did I say double-check? How about triple-check and quadruple-check? For very important surveys, you should:
- Self-Test – Run through the survey, answering it yourself, multiple times.
- Pre-Test – Invite coworkers or friendly outsiders to take the survey.
- Pilot Test – Invite 10% of the targeted list to take the survey.
- Publish – Invite the world to take the survey.
Self Test
Here are some of the things that I self-test or review before publishing a survey:
- Question flow: Do questions proceed in a logical manner from topic to topic?
- Question wording: Is each question worded clearly and unambiguously and is it free of typos and grammatical errors? The preceding question itself would make a horrible survey question: whenever one question asks for an answer on multiple, different topics, it should be split into multiple questions.
- Question types: Do the question types match the wording? Can respondents answer both "yes" and "no", or do the choices not correspond to the question? This frequently happens when a series of questions has the same choices or scale, where one or more questions is force-fitted into a method that doesn’t work.
- Scale consistency: Are related questions using the same scale? I just edited a questionnaire this week that had two different satisfaction scales, because one was written off the top of the survey author’s head and the other was copied from the question library. Do different scales arrange items consistently from best to worst or worst to best? I once had to ignore a question in analysis because it used a 1-5 rating scale with 1=best, 5=worst, in reverse of all earlier questions; the open-ended responses made it clear that some customers used the scale as written, and others used the scale as expected based on the earlier scales. (Yet another reason to avoid numbers in favor of labels in rating questions.)
- Answer validation: Is the survey configured to enforce the validation described in each question? For instance, making sure an email address is in the format jane@example.com, that that a fill-in-the-blank question is limited to numbers or that a choose-many question has a limit to the number of choices that can be selected
- Required answers: Are required answers used sparingly but appropriately, especially for critical questions and for questions that drive skip patterns? For closed-ended questions that are required, is there an appropriate choice in each case, such as “Don’t know”, “Can’t remember” or “Not applicable”?
- Skip patterns: Do the skips and conditional branches take respondents where we intended? Editing a question can sometimes delete or invalidate skip logic.
- Errors of omission: What questions did you leave out that you should have included?
Pre-Test
That last question is particularly hard to answer in a self-test. When I am feeling very unsure of a study, or the results are strategic rather than tactical, I will pre-test it on coworkers or, even better, on a small sample of the target audience (no more than 50). I will end the survey with some questions about the survey itself, to identify areas or survey structures that were confusing or ambiguous.
Pilot Test
Sometimes after a pre-test, I will pilot-test the survey to 10% of the participant list, in a “shakedown cruise” of what one favorite client describes as the “final draft but not the final final draft” of the questionnaire. This gives even more opportunities to catch errors before the survey goes live to the full list.
Publish
Ok, now you can publish the survey and invite one billion people to complete it. You still missed something—trust me. Most likely something related to one of your last-minute changes. But you’ve dramatically lowered your odds of missing something major.
Take it from hard-won experience: If your CEO or the CEO of your client cares about this survey, you definitely want to make sure you self-test, pre-test and pilot-test before you publish.
And that’s why Vovici surveys go into test mode first. Now fill up the tank and go on a test drive before that road trip.