Personalize Email Invitations to Improve Response Rates #aapor2010
Posted by Jeffrey Henning on Fri, May 14, 2010
At the 2010 AAPOR annual conference, Joel David Bloom, Ph.D., the associate director for survey research at the University at Albany, presented “Experimental Tests on Response Rates in Student Web Surveys: What Works and What Doesn’t”. Like many corporate researchers, Joel has a de facto panel of participants, a panel that is surveyed by other groups and that suffers survey fatigue over time. Improving response rates is a key concern for Joel.
Unfortunately, as a public university, the University at Albany has limited resources, and many proven techniques to improve response rate won’t work for him. Joel can’t offer the “Dillman Dollar” (a token $1 incentive for participation) as that would cost $12,000 for every study of the overall population. He doesn’t have a budget to send pre-notifications of web surveys by surface mail (something proven to work in other research) nor to use surface mail and telephone reminders. He does have the ability to send out emails—and more emails. “But we can’t send an unlimited number of e-mails because of survey fatigue, we don’t want surveys to overlap temporally, and we can’t poison the well.” His challenge, therefore, is maximizing response rates from the limited number of email invitations and reminders he can send.
Joel tested the following aspects of the email invitation process:
- Use of a pre-notification e-mail. Pre-notifications are effective when the pre-notification is in a different mode than the web survey itself (e.g., paper, phone). The same mode has not been shown to be effective, nor did it work for Joel. “Pre-notification was a waste of time. If you are sending an email to do a survey, put the damn survey link in it!"
- The source of the email in the “From” line. Joel split-tested an email sent from a high-status sender (in this case, the vice provost, who is well known to Albany students) and a lower status sender (himself!) and saw no significant difference in response rate.
- The content of the Subject Line. Joel feared that “the word ‘survey’ in a subject was a poison pill” and split-tested subject lines alternating “input” for the word “survey”. Again, he saw no discernible difference in response rate. [CMOR found dramatic differences for invites in their “A Playbook for Creating Survey Introductions for Online Panels”.]
- Personalization of the e-mail solicitations. A split test of personalization of a cable survey produced a 9.4% response rate for the personalized invite and a 7.7% response for the general invite (p=.009). For a student experience survey, the split test produced a 4.3% response rate for the personalized invite vs. a 3.1% for the general invite (p=.002).
Joel cautioned: “Caveats regarding external validity of these findings: University undergrads are different than other populations. UAlbany undergrads may not be the same as undergrads elsewhere.”
To support general applications of personalization, he referenced Dillman, Smyth and Christian (2009):
Social and behavioral scientists have long known that in emergency situations, the more bystanders there are, the less likely anyone is to step forward and help out…Although less dramatic, the goal of personalizing survey contacts is quite similar: to draw the respondent out of the group….Moreover, personalization can be used to establish the authenticity of the survey sponsor and the survey itself and to gain the trust of respondents, both of which should improve the likelihood of response. (p. 237)
Thus:
Personalizing all contacts in web surveys is important for the same reason as in mail surveys – it establishes a connection between the surveyor and the respondent that is necessary to invoke social exchange, and it draws the respondent out of the group. (p. 273)
While personalization is a powerful technique that Joel will use in future student surveys, he pointed out that it is inappropriate for some subjects and audiences. For instance, in an informal experiment with faculty, faculty recipients of personalized email invitations were concerned about confidentiality and anonymity.
Where personalization is appropriate, it can help produce higher response rates.