Welcome to the Listening Post!

Panels & River Samples Consistent in Behavioral Measurement

Tweet

 

While research has shown that different sample sources can lead to dramatic differences when contrasted with demographic benchmarks, Jamie Baker-Prewitt, a senior vice president with Burke, questioned whether such sample sources produce materially different results when measuring consumer behavior. In a presentation at the CASRO Panel Conference, Jamie reported that for the items that market researchers typically investigate - category purchasing, aided and unaided awareness, and brand purchase - differences by sample were relatively small across online access panels and river samples.

Jamie discussed the ramifications of the Yeager, Krosnick, Chang and Javitz paper, which concluded that non-probability online sample surveys are consistently less accurate than probability samples. Jamie said that unfortunately the RDD (Random Digital Dialing) approach used by the paper's authors is impractical and unaffordable for much commercial research. Since their paper focused on demographics rather than consumer behavior, Jamie's question was "Do different online sample sources evidence materially different patterns of consumer behavior?"

To answer this, Burke fielded a survey across two opt-in panels, two river samples and two social networking samples (derived from a social-networking aggregator and from ads placed on Facebook). The questionnaire looked at product category purchasing, brand awareness, brand usage and demographic variables. It looked at 15 product categories and 22 brands.

Jamie's key conclusion:

While some variation does exist in incidence of category purchasing, aided and unaided awareness, and brand purchase, the differences tend to be relatively small. Significant differences on omnibus tests were not found across the vast majority of consumer-behavior variables examined in this research. However, many of the differences that do exist involve the Social Networking Sample and the Facebook sample. Sometimes these two sample sources differed in the same direction, and sometimes they did not.

Caveats include:

• There were greater differences in samples drawn from Facebook and from a social-networking aggregator.
• Only high incidence items were researched. Differences for low-incidence products and services might be more substantial.
• Survey results were not compared to an RDD sample or Internet probability sample.
• No theory was presented for why panel and river samples might be representative for behavioral patterns when they are not representative of demographic benchmarks.