Optimize rather than Maximize Response Rates
Posted by Jeffrey Henning on Sun, Jan 03, 2010
My New Year's resolution to "send out fewer reminders" caught some readers by surprise. As it should have. Reminders are an effective way to increase response rates and, for decades now, maximizing response rates has been advocated as an important key to ensuring that survey research is representative of the target audience.
One advocate is AAPOR, the American Association for Public Opinion Research. While AAPOR does a phenomenal job of educating the U.S. public about the benefits and standards of polling, the organization is understandably conservative, and some of its advice is inappropriate for corporate researchers surveying their own customers and prospects. For instance, you have to love the wording of this suggested best practice from AAPOR: "Maximize cooperation or response rates within the limits of ethical treatment of human subjects."
Makes me want to found a new group, People for the Ethical Treatment of Respondents!
When those "human subjects" AAPOR refers to are current and potential customers, the last thing the researcher wants to do is aggravate them with repeated contacts simply to maximize the response rate.
First, despite the conventional wisdom, higher response rates do not necessarily produce more accurate results. The typical concern has been that low response rates indicate a potential non-response bias, in which people who were invited to take the survey but decline differ dramatically from respondents across the core issues being researched; as a result, the survey results are not representative. Only slowly has research emerged that begins to refute this. Pew Research Center found that a response rate of 25% did not materially change the quality of survey estimates when compared to the same study with a 50% response rate (see "Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey"). A number of papers have found only a small positive association between response rates and demographic representativeness, including Holbrook, Krosnick and Pfent in "The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms" [PDF]. That said, aim for response rates of at least 20%. Holbrook et al studied surveys with response rates from 4% to 70%, but four-fifths of the surveys had response rates between 20% and 50%, prompting the authors to observe, "Our findings seem most likely to apply to this range of surveys. Our findings might not generalize ... to declines in response rates below 20 percent."
Second, when surveying customers and prospects, their relationship with your organization is a paramount consideration. Sending three reminders to a survey about their three-minute call to your contact center can annoy customers more than it edifies you. Weigh how important the research you're conducting is and how strategic the decisions are that you will be making to determine if a focus on response rates is important.
Third, consider if a lower response rate might in any way bias the survey results. In the 1980s, I was involved with a landmark segmentation study of mobile professionals - we were surveying people away from their desk. In the age before ubiquitous cell phones, this meant high rates of callbacks - dozens of attempts to connect with potential respondents, to ensure that highly mobile professionals and corridor cruisers were accurately reflected in the study.
Before you relax your response-rate standards, conduct some analysis. Cross-tabulate responses received after reminder emails with responses received before - typically, you will find few statistically significant differences. When I did so for one project, I found no difference between people who responded to the second and third reminders and people who responded to the invitation or first reminder. For this particular study, I could have stopped at one reminder:
Who knows, you might find that you too can resolve to "send out fewer reminders" this year to optimize rather than maximize your response rate.