Representative Samples Are Attainable! Are Not!! Are Too!
Posted by Jeffrey Henning on Tue, Mar 02, 2010
Two of my favorite researchers this week painted extreme visions of my favorite research method.
What happens when The Unstoppable Force meets The Immovable Pundit? Well, in this case, I have to push back.
Before I leap to the defense of online probability samples, let me be fair to the prosecution:
- "There is no such thing as a rep sample. There are only good approximations of what we think a rep sample would look like." I think Annie's main point is that no technique is 100% foolproof and that some researchers are overly reliant on quantitative methods.
- When Ray says "online quant" he is talking about the majority of online quantitative research, which uses third-party panels. Ray states that online access panels have two key problems:
- First, they are not probability samples and therefore "online access panels are not representative of anything other than online access panels". (I would argue that, thanks to routers, online access surveys are no longer even representative of panels!)
- Only 1% of the population is taking 50% of the online access panels, meaning that the people being interviewed are fundamentally different than the people being studied.
I think both Ray and Annie are overgeneralizing. The research still indicates that random sampling, whether conducted online or by telephone, produces representative samples that can be matched back to demographic benchmarks. It is simply too expensive for most organizations to conduct such research, and online panel surveys are often good enough for general market research. (See Sample Quality of Online Panels: Putting Lipstick on the Piggy Bank.)
Meanwhile, more and more corporations are conducting more and more business online, capturing the email addresses of every single customer, prospect or user they interact with. Few today are empaneling this list but many are emailing the list and often getting optimal response rates. How satisfied were people with that customer service request or the online purchase? Corporations are answering such questions using a representative sample projectable back to their target audience.
To Ray's point of professional respondents, in a presentation yesterday, my co-presenter asked 80 college juniors if they were on a survey panel (after defining what a panel was and listing a few examples). Only one student was. Later I asked them if they were ever emailed and invited to take online surveys by companies they did business with: the vast majority of them raised their hand.
Online quant isn't busted, but it is shifting from access panels to in-house lists and is happening outside of the view of the Honomichl Top 50 market research firms.
Representative samples are 95% attainable, with a confidence interval of plus or minus two pundits.