Questionnaire Workshop: Refactoring One Question into Three Parts
Posted by Jeffrey Henning on Thu, Feb 26, 2009
A customer writing their first questionnaire sent me a question like this [edited for confidentiality; published with their permission] and asked for help with it:
Please rate each feature using the following scale.
- 4--I'm confident in my ability to use this feature/can use without assistance
- 3--I'm familiar with this feature but ask for help from time to time
- 2--I know about this feature but am not confident in my ability to use it
- 1--I didn't know about this feature
- 0--I know about this feature but use the mainframe source for this information rather than the new web interface
| Account holder profile |
0 |
1 |
2 |
3 |
4 |
| Billing information |
0 |
1 |
2 |
3 |
4 |
| Opportunity tracking |
0 |
1 |
2 |
3 |
4 |
| Product usage |
0 |
1 |
2 |
3 |
4 |
| Service subscription |
0 |
1 |
2 |
3 |
4 |
| Support ticket log |
0 |
1 |
2 |
3 |
4 |
Here are some of the problems I identified:
- It will be difficult for the respondent to remember the scale.
- The numbers don't convey any useful purpose for analysis.
- The choices in the scale are not mutually exclusive: for instance, someone might know a feature and be uncomfortable with it (#2) as well as use the mainframe source (#0).
- The scale is trying to measure too many different things: knowledge about a feature, need for help using a feature and whether or not the mainframe is preferred.
While questionnaires should be as short as possible, when one question covers multiple different aspects, it is best to break it into multiple questions.
Matrix questions can be imposing to respondents and tedious to complete. To counteract this:
- I would configure the list of topics in the matrix question to only show those choices that the respondent checked in the prior question. Therefore most respondents will not see all six rows, but only the ones that are relevant to them.
- For a two-sided matrix, I want to keep the choices as few and simple as possible, hence the use of a common three-point choice list ("Never", "Sometimes", "Always").
Writing questionnaires is like writing blog posts or white papers or web pages - practice makes perfect, which approach is better than another can be subjective, and there can be as much art as science to it.