Welcome to the Listening Post!

Response Rates Driven by 16 Major Factors

Tweet

 

Weimiao Fan and Zheng Yan have published a comprehensive paper in the journal Computers in Human Behavior (March 2010) that synthesizes the results of 300 studies into a model of the contributing factors of response rates. While some researchers use response rate to refer to the number of people out of the target audience who start a survey, Fan and Yan use it, as others do, to refer to the number of people who complete the survey divided by the number of invited potential respondents.

The article "Factors affecting response rates of the web survey: A systematic review" provides a thorough summary of the many factors that can increase or decrease response rates.

Target audience

• Type of population: "General populations are found less willing to respond than employee populations, student populations, or army populations)." [See Survey Response Rate Directly Proportional to Strength of Relationship.] Within business populations, senior managers are less likely to respond than middle managers and staff.
• Demographics: Response rates vary by age and race, even after controlling for Internet access and computer literacy.
• Personality: Respondents are more likely to be conscientious, agreeable and open to new experiences than non-responders. Respondents "with a higher need for cognition" are more likely to start a web survey but less likely to complete it; respondents "with a higher emotional stability" are more likely to complete the survey.

Content of web questionnaires 

• Nature of the sponsors of the survey: Government and academic sponsors get higher response rates than businesses.
• What the topic is: Subjects that are of great interest to respondents have much higher response rates. Surveys on sensitive subjects have lower response rates.
• How long the survey takes to complete: "The length of a survey generally [has] a negative linear relation with response rates." 
• Question writing: Poorly worded questions and technical issues cause respondents to exit surveys.

Survey delivery

• Sampling methods: Unsurprisingly, panels provide higher response rates than email lists used for one-off invitations to surveys [because panelists have in effect agreed to participate in surveys].
• Contact delivery modes: Contacts include pre-notification of the upcoming survey, actual invitations and reminders. Email contacts reduce response rates when sent to abandoned email addresses or when blocked by spam tools. Research is inconclusive on whether increasing the number of delivery modes (for instance, sending SMS pre-notifications and sending non-responders a hard copy of the questionnaire) increases response rates.
• Contact quantity: "Several meta-analyses of both mail and web surveys have consistently concluded that the number of contacts is one of the most important factors to predict response rates." [The authors cite some circa 2001 research into the value of pre-notifications, but more recent research I've seen shows no effect for pre-notifications.]

Invitation designs

• Personalization: Some types of personalized email invitations improve response rates while others (e.g., personalized greetings, personalized email addresses) fail to. As a side effect, personalization does increase the risk of social desirability bias.
• Scarcity: Mentioning to recipients that they were part of a small selected group to be chosen to participate in this research increases response rates.
• Access control: Of three types of logins tested (username and password, only a password, or an automatic URL with a unique identifier embedded), the password-only login resulted in higher start rates, higher response rates and less socially biased answers.
• Use of incentives: Adding incentives did not significantly improve response rates, according to two studies (Goritz, 2006; Porter & Whitcomb, 2003b). [I think this varies significantly based on the type of survey and the relationship between the sponsor and the targeted respondents.]

Technology issues

• Survey software: Incompatibilities between survey software and different browser configurations [especially mobile browsers] can lead to reduced response rates. "Because of these variations, some respondents may not be able to...submit their answers successfully."
• Internet transmission capabilities: Intermittent Internet access can also lead to reduced response rates [again, especially a concern with mobile surveys].

This paper serves an important role in inventorying the many factors that affect response rates to web surveys. Tune into tomorrow for practical suggestions on how to improve the response rates to your own surveys.

Update: Improving Response [Start] Rates to Web Surveys (3/17).