Commercial Online Panels from a Total Survey Error Perspective
Posted by Jeffrey Henning on Mon, Mar 29, 2010
AAPOR (the American Association for Public Opinion Research) has just published an excellent review and evaluation of commercial online panels (that is, panels that third parties can pay to field surveys to). Under the able leadership of
Reg Baker of Market Strategies International, a special task force has evaluated the state of such panels today.
Using the framework of Total Survey Error, the task force found the following issues with research using commercial online panels.
Respondent Selection Errors
- Sampling Error - As AAPOR has published before, "the reporting of a margin of sampling error associated with an opt-in or self-identified sample (that is, in a survey or poll where respondents are self-selecting) is misleading." Non-probability panels are subject to sampling error, but the range of that error cannot be calculated.
- Coverage Error - For U.S. panels, the lack of Internet usage in approximately one third of households means a large group is excluded from the sample, with certain subgroups excluded disproportionately (e.g., non-whites, the elderly, the poor, the rich, the unmarried, the less educated, those with less civic involvement). For those seeking to use panels to represent the Internet-using population, panelists differ from other online users in being higher educated and may participate only in surveys of greater topical interest to themselves; further, recruitment strategies for non-probability panels may introduce coverage error for specific target groups: for instance, by partnering with specific web sites with different Internet audiences. [While not explicitly mentioned in the report, routing bias may cause coverage errors for panels that use priority routers. This error source is especially troublesome for those seeking for replicability within a panel.]
- Unit Nonresponse Error - The self-selection bias inherent in recruitment to open non-probability panels is a source of error: nonresponse during panel recruitment is enormous, with one study (Alvarez et al., 2003) having only 6% of click-thrus complete the full panel registration process for only one successful panelist signup per 5000 banner ad impressions. Probability-sampled panels avoid this source of error by using detailed external selection criteria with a rigorous and extended process to convince identified households or individuals to join the panel.
Response Accuracy Issues
- Item Nonresponse Error - The report did not discuss item nonresponse error, which may differ materially from empanelled respondents vs. non-empanelled respondents; this issue has not yet been studied.
- Measurement Error due to Respondents
- Incentive maximization leads some panelists to over-report product ownership and business purchasing authority to qualify for more surveys. It leads other panelists to create multiple accounts within a panel.
- Panel conditioning, deriving from the repeated completion of surveys, may change the attitudes and behaviors of some panelists, especially in regards to purchase intention and brand awareness. The good news is that such conditioning may reduce satisficing and improve the quality of answers.
Survey Administration Issues
- Postsurvey Error - The weighting of data derived from online panels has enabled marketing research firms to differentiate themselves from one another. Unfortunately, the academic literature has yet to validate a weighting scheme for general market research. Only a few narrow domains, such as political polling, have demonstrated greater accuracy from using weighting.
- Mode Effects
- Social desirability bias appears to be reduced in online research. [AAPOR was too conservative here; the studies prove this to my satisfaction.]
- Satisficing is much less prominent online than offline.
- Compatibility Effects - The same questionnaire fielded to multiple nonprobability panels often shows wide variances in answers, as the recruitment and quality practices differ significantly from panel provider to panel provider.
As AAPOR has recommended before, the following wording should be used when documenting surveys with non-probability samples:
Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate in (company name) online surveys and polls]. The data (have been/have not been) weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel] rather than a probability sample, no estimates of sampling error can be calculated. All sample surveys and polls may be subject to multiple sources of error, including, but not limited to sampling error, coverage error, and measurement error.
So when should you use commercial non-probability panels? The task force concludes that they should not be used for estimating population values but can be used to:
- Determine the interactions of attitudes, behavior and intentions within personal attributes
- Contrast the receptivity of different target markets to product concepts
I would argue that commercial online panels are perfectly acceptable whenever a convenience sample is appropriate. Additionally, they are satisfactory for many businesses to use when studying Internet behavior and potential opportunities for Internet services, provided a panel is chosen whose recruitment strategies minimize coverage error for the target audience being studied.
For more information, download this PDF of the AAPOR Report on Online Panels.