Causes of Survey Incompletes: Why Panelists Say They Abandon Surveys
Posted by Vovici Blog on Tue, Feb 22, 2011
At this year’s Online Research Methods conference, James Sallows, vice president of EMEA client operations with Lightspeed Research, showed a slide with a crosshair shooting the logos of Lightspeed, Toluna, SSI, GMI and ResearchNow. If he feels a bit defensive, it is because much of the debate on the quality of panel research has centered on the quality of the panelists, and thereby, the quality of the panel providers themselves. James argued that just as important, maybe more important, is the quality of the surveys being fielded to the panelists.
As a point of illustration, in one analysis that Lightspeed conducted, they found that in three months of surveys fielded to their panels, 50,000 respondents were “bad”. Yet, when they changed the unit of analysis from respondent to survey, Lightspeed discovered that half of the surveys had not a single bad respondent. “We have been looking at the data quality issue, and it is not just a measure of well-behaved respondents. We must address the surveys themselves and better understand how to keep respondents engaged. Let’s start with incompletion rate, which is the simplest survey-level indicator of engagement and a gauge of quality on every project.”
What are the drivers of incompletion, the making of unhappy respondents?
In one analysis that Lightspeed conducted, 35% of incompletions were because of the subject matter of the survey, 20% were due to media downloads, 20% due to survey length, 15% due to grids and 5% due to too many open-ended questions. Here’s what James had to say about each in turn:
- Subject matter – “Fabric conditioner is a dull subject matter, and it was an issue [for incompletes] early on in the questionnaire.” Automotive surveys, on the other hand, have a lower rate of incompletion compared to most other subjects; respondents are engaged with the topic.
- Media downloads – Older respondents won’t wait for long download times; 18- to 34-year old respondents, on the other hand, are more willing to wait while media is downloading.
- Survey length – Many respondents abandon the survey at the introduction, then remain, with a steady increase in incompletes after 15 minutes.
- Grids – One client frantically called James into a meeting due to problems with “bad” respondents. They said, “Something has to be done! Look at the data – respondents are flatlining after the 35th grid!” Even relatively small grids seem like a lot of work to respondents, let alone grid after grid. “Making grids mandatory pushes people to drop out – there is a selection of people that don’t want to do them.”
- Open-ends – Having too many questions that require respondents to type in answers leads to greater incompletion.
Only one of James’ 800 clients has an internal gate-keeper who validates the quality of surveys before they go to field. Many questionnaires are simply not up to research standards and have not paid any attention to respondent engagement.
James summed up by saying, “Online research is only sustainable if the industry works in partnership to address both panel and survey quality concerns. Respondents are not a limitless resource – we need them to want to take more questionnaires. Keep a constant eye on the incompletion rate of your own surveys and take action: your data is ultimately the victim of a survey that does not engage respondents. It’s all about engaging the audience – it’s good for the respondent, it’s good for the panel and it’s good for your data.”
See also: