Welcome to the Listening Post!

Sample Blending

Tweet

 

Peanut Labs invited me to moderate a webinar this afternoon on "To Blend or Not to Blend ... That is the Question." The speakers were Steve Gittelman of Mktg Inc., Jackie Lorch of Survey Sampling International and John Bremer of Harris Interactive.

Sample blending is the science of producing representative results when using multiple sources for sample. Each speaker discussed their own proprietary approach to this problem.

Correcting Blended Samples to a Baseline

John Bremer, who pioneered the proprietary propensity score weighting technique that Harris uses, provided some background: One of the major findings of the ARF Foundations of Quality research was that "different sample sources may produce different results even after the demographic characteristics are equalized." John felt that reaction to the results were overblown, but the implication was that "switching between samples or different blending proportions could mean different results."

Whether or not you need to blend depends on context. John proclaimed himself "agnostic" about blending, finding it fit certain circumstances and was unnecessary in others. One area where it is important is in tracking studies. "Are differences due to shift in sample or real changes in the marketplace? What if no differences are observed when they should have been? Are comparisons to established norms resulting in correct decisions if not the same sample composition?"

He gave an example of ad testing, where norms had been carefully developed from a single sample source. The amount of research now being required meant that the original sample was no longer sufficient; since go/no-go decisions on launching new ads was sensitive to small changes in the results, it was imperative that the new sources be carefully blended to provide consistent results with the original source.

John presented his basic philosophy on correcting blended samples to a baseline.

• Detect - Is there a difference?
• Calibrate - What would correct the difference?
• Estimate - Apply the correction.
• Validate - Check to see that things are how you think they should be.

1. Identify potential variables
• Must be correlated with outcomes of interest that differ across samples.
• Must include demographics but go beyond demographics as well. 
2. Run regression to determine which variables are likely to contribute to differences
• Dependent variable is indicator of which sample respondent comes from
• Independent variables are key variables from Step 1
3. Weight one sample to look like the other sample on variables found to be key.
4. Do it again until nothing new is found
   • Must include demographics while also going beyond demographics for solution
   • Demographics can be easily solved. Usually the answer is beyond demographics

Indentifying Underlying Factors to Blend By 

Jackie Lorch discussed her recent SSI research into factors that can be used to correct for multi-source samples. Blending might have been a "necessary evil", a measure of last result, but it is increasingly becoming simply "necessary". More people golf than are members of country clubs, and more people aren't on access panels than are; sample blending lets you incorporate river samples (intercept surveys) into your survey research.

Why do sources differ? Jackie gave the example of a sample derived from a fishing website. For many questions - coffee consumption, soda preferences - such a sample would be fine. But if anglers are away fishing every weekend their media consumption may differ dramatically from nonanglers. How do you correct for such differences?

Traditionally, surveys have been weighted demographically, yet SSI found that varying demographic weights had little effect on overall results. To find better weights, SSI analyzed 162 factors, including personality traits, music preferences, cognitive types, neurographics, propensity factors, geographic/personality alignment, chronotypes, social values and more. These were then tested against questions on technology, hobbies, interests, brand preference, loyalty and ad awareness: the issues most frequently covered in survey research. The psychographic factors explained more variance than demographic factors alone. For instance, ownership of Blackberrys and similar devices correlates to level of adoption of new technology which correlates to willingness to try new things: this underlying attitude is a more important predictor than age, gender or education.
SSI is still working on the perfect mix of factors and is looking to develop a half-dozen questions that can explain most variance and be used to better blend samples.

Blending by Individuals

Steve Gittelman of Mktg Inc., who published The Grand Mean Project on online panels, said that "We can blend in two ways: The first is by understanding groups and the second is by understanding individuals. We measure groups through subsamples. In contrast we could measure all the individuals of a group and then blend them for each study as it comes along, one at a time. The best blends are where we accumulate data one respondent at a time."

Like the other speakers, Steve said that demographics is but one layer - for Steve, behavioral segmentation is the crucial second layer. Steve's blends are based on 30+ variables drawn from the profiles of panelists. Depending on the study, sample can controlled for different levels of respondent experience, purchasing intent, media consumption and other behaviors.

Unlike John, Steve believes that sampling blending is now critical to every study, because a single sample source itself should be regarded as a blend, with its own mixes of different segments of people.

Interested in learning more? Here is a copy of the Sample Blending presentation slides, courtesy of Peanut Labs. And see Annie Pettit's post, To Blend or Not to Blend, That is the Question.