What's the Catch? Does Sample Sourcing Matter
Posted by Jeffrey Henning on Thu, Jun 12, 2008
At the MRA Annual Conference last week, Melanie Courtright, a vice president with DMS Research (an AOL subsidiary), presented the results of a research study she designed contrasting the results gathered from telephone surveys, Internet-panel surveys and Internet-river surveys.
So-called "river surveys" dip into the stream of users visiting thousands of web sites to invite them to take a survey, rather than emailing a pre-built panel of respondents. DMS's Opinion Place river runs promotions across thousands of sites, invites visitors to take a survey, then asks them an 11-question qualifier which helps route each visitor to a specific survey by their demographics. Her title, "What's the Catch?" alludes to catching these respondents from the river. (As she described the Opinion Place river, I had to laugh as I envisioned a catch-and-release program for respondents, where they were released back to the Internet after completing their survey!)
In Phase I of the research, conducted in December and collecting 2,412 responses, phone, panel and river surveys were contrasted. The results uncovered minor attitudinal differences but uncovered significant differences in technology usage and adoption, as would be expected. For instance, 69% of telephone respondents had been using the Internet for at least 5 years, compared to 80% of the river respondents; 22% of phone respondents don't have home Internet access, compared to 1% of the river respondents; 21% of phone respondents don't have a cell phone, compared to 9% of the river respondents.
An additional key area of difference was in the amount of survey participation. Only 4% of phone respondents take surveys weekly, compared to 17% of river respondents and 70% of panel respondents. The average number of surveys per month was 0.3 surveys per phone respondent, 2.9 surveys per river respondent and 16.6 surveys per panel respondent. Clearly, for researchers worried about the affect of professional respondents on results, telephone or river surveys are better choices than third-party Internet panels.
A Phase II study of 3,647 responses was conducted in April across seven panels. It found surprisingly few differences between results derived from specific panels, with the difference again being technology usage, as some panels seem to have more early adopters than others.
For data quality, 21% of panel respondents completed the questionnaire in 7 minutes or less, compared to 5% of river respondents. DMS found quality problems such as answering a trick question wrong or not answering an open-ended question thoughtfully were far more typical of speeders. Melanie said, "If there is one step you take to improve the quality of your responses, take a look at the responses from speeders."
When mapping the survey results to standard benchmarks such as percent of the U.S. population that are married, are students, are Sunday newspaper readers, etc., each survey methodology produced results close to the benchmark. However, each methodology produced very different results on technology usage, again as would be expected given that the methodologies differed by technology.
A pointed question from the audience said that probability sampling was the theoretical basis for the projectability of survey research and asked what the scientific underpinnings were for assuming that Internet research was similarly representative. Melanie answered that replicability is emerging as the standard instead of randomization and that the results from her research were replicable.
Answering the question posed by her presentation title, Melanie echoes Brad Bortner at Forrester to say that sample source doesn't matter as much as people may think. Melanie's recommendation was that third-party panels were best for surveying low-incidence populations, phone surveys were best when concerned about the skew from online methodologies, and river surveys were best for reaching less surveyed Internet populations.