Forrester on Representativeness of Market Research
Posted by Jeffrey Henning on Sat, May 31, 2008
Brad Bortner, a principal analyst with Forrester, has written an excellent white paper about the representativeness of online research. Here's an abstract of Does Declining Research Projectability Matter? (emphasis added):
"Conventional wisdom holds that online panels are inherently problematic in supplying projectable findings to the general population - online and offline. Critics assert that online panels aren't representative and that new types of dirt in the data - such as professional survey takers - erode the accuracy of projections. However, it's hard to find any substantive marketing decisions that have gone awry due to using online panel-based research. Why? Because good online panels actually deliver as much accuracy as they give up. Market research professionals should leverage what online panels are good at - speed, cost, and new analytic approaches - and at the same time hedge risk by mixing data collection modes, expanding their analysis framework to create 'what if' models, and understanding that sometimes speed of result trumps accuracy and cost."
Brad writes about third-party panels, which he sees as sufficient for research but suffering from three key problems:
- Wide variances in projectability from different panels
- Underrepresentation of key groups
- Bias introduced by "professional" survey takers
Because proprietary panels are derived from company databases, proprietary panels in contrast are highly representative of the target population of a company's clients and employees. See this blog post last month for a more detailed comparison of the two types of panels: Panelist Quality of External Lists vs. Feedback-Focused Communities.
If you have any concerns about the projectability of online research, I strongly encourage you to buy Brad's report.