Survey Testing Case Study
Posted by Jeffrey Henning on Tue, Aug 25, 2009
Brian’s post on our
CE IQ survey reminded me of the
survey test and review process I went through for that questionnaire. Here are the types of edits I made at each of the four levels of testing:
Self-Test – “Run through the survey, answering it yourself, multiple times.”
I spent five hours reviewing the questionnaire. Each time I read it, I found a new issue. My goal was to get to the point where I would self-test it and not find a single issue. Some of the items I encountered:
- Upon re-reading, I found quite a few questions were worded awkwardly. I did lots of proofreading and wordsmithing to make sure questions were clear.
- I encountered a few double-barreled questions that I split into separate questions.
- I only had three skip patterns configured but upon testing discovered I had specified two of them incorrectly!
- I discovered that I didn’t have no-opinion choices on the required questions.
- I realized one rating scale was backwards from the others (choices listed from most favorable to least favorable where the rest were listed with negative choices first).
- I wanted to make sure each choose all that apply question had an “Other” choice in the list.
- I noticed things that would only bother a copy editor or someone, like me, who had done copy-editing in the past. For instance, this type on inconsistency bothered me: ending choice lists with “Other, Please specify:”, “Other (please specify)” and “Other, please specify” rather than one consistent phrase.
- On the last pass, I realized that I had asked for firmographics but no demographics.
Eventually I gave up. There was one choose all that apply question where two of the choices were mutually exclusive – that question really should have been reworded, but I noticed it on my last pass through the questionnaire and was far over my time budget for this phase of testing.
Pre-Test – “Invite coworkers or friendly outsiders to take the survey.”
The great thing about doing a survey with others is that the more eyes the more likely you are to catch problems. The open-source movement has an axiom for this: "Given enough eyeballs, all bugs are shallow” (also known as
Linus’s Law).
The bigger issue that was noticed was that the questionnaire consisted of a mix of British English and American English, which made it annoying to both Americans and Brits.
CGA, one of our partners in the UK, wrote the first draft of the questionnaire, and I wrote the subsequent drafts. To clean it up and make both sides of the Pond happy, I ended up fielding the survey in U.S. English with a U.K. English translation (e.g.,
organization to
organisation,
dollars to
pounds sterling, etc.).
The pre-test also caught a minor font issue (serif in tables, sans serif elsewhere) that had broken a customized display theme.
During the pre-test, we made sure to test the questionnaire in three browsers (Internet Explorer, Firefox and Chrome).
Pilot Test – “Invite 10% of the targeted list to take the survey.”
I’m always nervous about sending out an invitation to my entire list of potential respondents, in case I realize one click too late that I’ve made a mistake. I sent the initial survey invitation out to 10% of the first wave of invitations, and everything went fine.
Publish – “Invite the world to take the survey.”
Once it was clear that the survey was running smoothly, I invited everyone on the first wave of my mailing list.
Was I happy to end up spending over 8 hours testing a fairly straightforward 35-question survey? No, but I owed it to my hundreds of respondents to make the survey as streamlined and simple as possible.