How to Maximize the Number of Open-Ended Responses
Posted by Jeffrey Henning on Thu, Sep 24, 2009
My standard advice about structuring questionnaires is, after the screener, to begin the questionnaire proper with open-ended questions. Since respondents start fresh and then tire over the course of the survey (moving from giving the optimal answer to satisficing), you are going to get more feedback when you lead with these open-ended questions. Further, starting with open-ends removes the chance for earlier closed-ended questions to bias the answers to open-ends, as they raise other topics to consideration than what might have been top of mind for respondents.
Many of my customers push back on this advice, however, because they are concerned that leading with verbatim questions will increase the survey abandonment rate. They prefer to keep their open-ended questions near the end of the survey.
I recently had the opportunity to put my advice to the test. I did two versions of a 5-page, 9-question questionnaire. Survey A had the open-ends as the seventh and eighth questions. Survey B moved the same questions to the first and second position; everything else stayed the same. Respondents were not required to answer the open-ended questions; clicking the Next button would simply take them to the next question.
The test was fielded in August. Survey A had 70 responses from 290 invites (24% response rate), and Survey B had 79 responses from 379 invites (21% response rate) [had the test been the primary purpose of the survey, I would have kept these distributions equal].
As expected, the abandonment rate did increase when leading with open-ended questions, climbing from 1% for Survey A to 6% for Survey B. Yet that trade-off was worth it: 92% of respondents answered at least one of the open-ended questions for Survey B, compared to only 60% of the respondents of Survey A. So in exchange for a modest drop-off in overall response, we gathered fully 50% more open-ended comments!
Surprisingly to me, though, the abandonment rate for Survey B did not occur at the beginning of the survey-I fully expected respondents to exit without answering the open-ends, but no respondent did so. Instead what happened was that respondents who answered at least one of the open-ends were the respondents who were more likely to grow tired of the survey and abandon it. In fact, every respondent who skipped answering the two open-ended questions answered all seven subsequent closed-ended questions.
I had also hypothesized that you would get longer answers when leading with open-ended questions. In this, I was disappointed: the average length of an answer was 13 words in both surveys.
As a result of this test, I will continue to advise that open-ended questions are more productive when placed at the beginning of a survey rather than near its end.