Respondent Satisfaction: Post-Survey Assessments
Posted by Jeffrey Henning on Mon, Apr 12, 2010
According to our research study, Lessons from Professional Panel Providers, 59% of panels administer post-survey assessments. Given concerns about survey length, it seems counterintuitive to lengthen surveys by asking additional questions about the survey experience itself. This is not a practice that most managers of proprietary panels need to emulate, but we can learn from research that leverages this practice.
PROOF
For instance, PROOF has an online panel of Croatian Internet users. Panelists are commonly asked these questions (in Croatian):
- What do you think about the length of this questionnaire? Too short, optimal, somewhat too long, absolutely too long
- How interesting did you find this questionnaire? [rating scale]
- Is there something Proof could do to improve the satisfaction of its members?
- What do you think about Proof in general?
By analyzing the answers to these questions, Mirta Galešic found that interesting questionnaires are perceived as shorter questionnaires (c.v., Perceived Questionnaire Length).
Research Now
Some panel management tools, including that of Research Now, redirect the respondent from the survey back to the panel portal to complete the assessment of the survey. Research Now asks the following five questions, using a
bipolar 5-point agreement scale:
- The survey was interesting
- The survey was easy to understand
- The survey was user friendly
- The reward was fair compensation for taking part
- I will probably take part in more surveys similar to this
A final open-ended question prompts for comments about the survey.
The five agreement items are mapped to a scale from 0 (strongly disagree) to 100 (strongly agree) and then averaged for the panelist's survey satisfaction score. Research Now found that satisfaction was negatively correlated with duration of a survey in minutes: respondents were less satisfied with longer surveys. More interestingly, Research Now was able to demonstrate the rate at which less satisfying surveys decrease the participation rate in future surveys.
OMI
A few panels provide discounts to customers whose surveys are rated by panelists as significantly above average in satisfaction when compared to all surveys fielded to the panel. OMI, for instance, offers a 5% discount for studies that score in the top 25% of panelists' satisfaction ratings.
Norstat
A key reason to measure respondent satisfaction is so that you can work to improve it. The challenge is that the very dissatisfied won't respond - they will abandon the survey earlier. The answers, therefore, are representative only of respondents who completed the entire survey.
Despite that caveat, if your organization is suffering from low response rates, these questions can help you better understand respondent satisfaction. Dan Kvistbo, the online director for R&D at Norstat, offers this advice:
I've been tracking best practices on this lately and have collected a variety of approaches from different agencies, some quite extensive, others limited to a few scaling questions. In my view, the trick here is to keep this as short as possible and as long as necessary. That is, to balance the extra "burden" on the survey participant - with the relevant satisfaction dimensions you wish to include and intend to use!
Respondent Satisfaction Template
If your organization is concerned about declining response rates and believe that the quality of the surveys themselves is a major part of the problem, I would encourage you first to examine the questionnaires you are using to see what if they are following best practices for maximizing completion rates. If they aren't, then use questions like the following to measure survey satisfaction:
- How understandable were the survey questions? Not at all understandable, slightly understandable, moderately understandable, very understandable, completely understandable
- How interesting was the survey? Not at all interesting, slightly interesting, moderately interesting, very interesting, extremely interesting
- How likely are you to take the next survey we invite you to? Not at all likely, slightly likely, moderately likely, very likely, completely likely
- How could we have improved this survey? [open-ended]
Again, extend the questionnaire in this manner only as a last resort.