Random Thoughts on 2008 MRA Annual Conference
Posted by Jeffrey Henning on Mon, Jun 16, 2008
This post wraps up my coverage of the Marketing Research Association's 2008 Annual Conference with some random thoughts and some thoughts on randomness. My past MRA posts:
For me, the common themes I heard across the sessions I attended were concern about data quality, concerns about third-party Internet panels in particular and interest - and trepidation - about online communities.
The concerns about data quality were many, and for this I do blame the Internet. The rise of inexpensive survey software, an application category I pioneered, has certainly contributed to an erosion of the perceived value of surveys. Because web surveys can be conducted so inexpensively, many end users are not willing to spend the amount required for the highest level of data quality. If this were an informed tradeoff on the part of research users ("yes, the decisions that will be made from this study aren't that important, so let's not spend a lot" or "the decisions made from this research are critical to the future of the organization and therefore worth spending extra time and money to get right"), that would be acceptable, but as one attendee joked in the Future of MR session, "the one thing you never hear from a customer is 'Take your time'."
Besides the Internet placing price pressure on all forms of survey research, it has also created an expectation that research can and should be conducted rapidly. Robin Pearl of Estée Lauder related a humorous anecdote in her keynote. A salesperson for a panel company excitedly told her how research would be turned around in hours in the middle of the day. To which Robin replied, "Well, I guess I won't be using your company then." The salesperson didn't understand that conducting an entire survey in a few hours would not be representative of the U.S. population, who are not yet chained to their email 24 hours a day but have usage that varies by hour of day and by time zone.
I was not surprised at the backlash against third-party Internet panels, because the panel providers have been proactively working to address many of the research industry's concerns about panel quality. I was surprised at the faith in random-digit dialing. I do believe there was this Golden Age from the 1950s to the 1990s where randomly composing a phone number and calling it provided a true probability sample. But I do not believe that RDD today provides a random sample of the U.S. population. With its exclusion of cell phones, it now underrepresents the youngest adults and overrepresents the oldest adults, and it underrepresents early adopters of technology and overrepresents laggards (as DMS Research showed in their presentation).
That said, without probability sampling you can no longer estimate sampling error, and you can no longer assume that your sample accurately reflects the target population. The willingness of many Internet researchers to completely ignore this and to treat convenience samples as representative does the industry a disservice.
But I did not hear a single speaker reference the fact that for an increasing number of businesses Internet surveys can in fact provide random samples! More and more businesses are moving online: a favorite bookstore of mine in Boston now reluctantly sells only through the Web, and many other ma-and-pa shops have moved from Main Street to a Uniform Resource Locator. All such companies maintain email databases of their past and present customers, or - for those who provide Software as a Service - their past and present users. For them, a random sample is an email campaign away. They can even schedule a series of reminder invitation emails to reduce nonresponse bias. They get the benefit of rapid and inexpensive Internet research with the statistical foundation that probability sampling provides.
That meets their quantitative needs. For their qualitative needs, online communities are the next major trend in Internet research. Whether end-user organizations build their own community using software such as Vovici EFM Community Builder or partner with a research firm such as Intrepid Consultants, organizations can gather more qualitative information more rapidly than ever before. The trend is new enough that we can't even decide on what to call this tool - I heard the terms online portal, private online community and advisory council used. Whatever we call it, it presents a new research resource that can supplement and sometimes even replace traditional research that is becoming less effective.
So that's my take on this year's MRA Annual Conference. Next up- November 3-5th, MRA's 2008 Fall Conference in Las Vegas, where those of you who really love randomness can bet on it.