Welcome to the Listening Post!

Convenience Samples: Pros and Cons

Tweet

 

We survey samples of a target population when we can’t afford to survey every single member of that population. Face it: censuses are expensive. As Robert Groves shared at the MRA First Outlook Conference last week, while it costs 42 cents for a mailed back U.S. Census form, it costs $57 to obtain Census information for each household that does not mail the form back!

If we want to project from the results of a survey to our target audience with a knowable margin of error, we use random or probability sampling, which provides for equal opportunity for selection, with external selection of any member of the target population. When we can’t use a probability sample, we may have to take what respondents we can get: a process known as convenience sampling.

The advantage over random samples:

• Convenience samples are cheap. Mail a survey invite to your house list; post a link to your website, Twitter or Facebook; rent an online access panel—that’s a convenience sample. And they’re affordable, often simply taking time to leverage existing assets. Even if you do want to rent a list, it is much more affordable than a probability sample. For one recent study we quoted, it cost $800 to rent a list or $4,400 to field the same survey to a random telephone sample; quite a difference in cost!

The disadvantages:

• Convenience samples do not produce representative results. If you need to extrapolate to the target population, convenience samples aren’t going to get you there.
  • For example, there was the 2008 AOL poll with 272,939 votes in which 61% of respondents voted for John McCain for U.S. President and 39% for Obama. Much larger convenience samples are not more accurate than small probability samples.
  • At the 2009 CASRO Data Collection Conference, Jon Krosnick compared the results of a telephone probability sample to that of seven convenience samples conducted via the Internet. Where 54% of respondents said that they had seen a movie recently in the probability sample, the convenience samples varied widely, with answers ranging from 65% to 93%. Weighting the results of the convenience samples didn’t bring the answers into line with the probability sample.
• The natural tendency is to extrapolate from convenience samples. The tendency when using convenience samples is to treat the results as representative, even though they are not. Many people do not understand the theoretical underpinnings of probability sampling and treat any survey results as accurate representations of the target audience. While mainstream media outlets often will not publicize the results of surveys that used convenience samples, small media organizations often will, without describing the methodology as a convenience sample.
• The results of convenience samples are hard to replicate. If you analyze the results of a convenience survey by list source, you will often find dramatic differences in the answers from the different lists, often in ways that confound easy explanation.

Getting representative results – results that can extrapolated back to the target population – is not always a research objective. Surveys fielded to convenience samples have many of the advantages of surveys in general, which is why the sampling technique is so widespread.

• Convenience samples can be used to intervene to satisfy dissatisfied customers. A key, often forgotten aspect of probability sampling, is its dependence on external selection: inviting and then repeatedly reminding people to take a survey, which helps ensure representativeness. Putting a survey postcard with every bill presented at a restaurant is a convenience sample, since there is no followup and encouragement to take the survey: no true external selection. And in such cases dissatisfied customers are often more likely to complete such surveys – the survey does provide an opportunity to hear from such customers and ask them for contact information in order to take action to improve their satisfaction.
• Convenience samples can provide rich qualitative information. When illustrative quotes are important, surveys to convenience samples can be a great source of rich verbatim comments on specific topics. The survey can also provide detailed demographic profiles to shed further light on the comments.
• Convenience samples may provide accurate correlations. Some argue that correlation research is accurate enough with convenience samples, since the study is not of proportions of the target audience but of the relationship between variables.

The decision to use a convenience sample instead of a probability sample is often driven by cost. That’s fine, and an appropriate trade-off in many cases; just don’t make the mistake of assuming that you can extrapolate from that convenience sample back to your target population.  

See also:

• Recommended Sample Size for Accurate Surveys
• Sample Quality of Online Panels: Putting Lipstick on the Piggy Bank
• Total Survey Error: Approaches for Minimizing 9 Common Issues