Quality - What's Love Got To Do With It?
Posted by Jeffrey Henning on Mon, Jun 09, 2008
I attended the Marketing Research Association's Annual Conference last week in New York City. The MRA Conference is a great opportunity to gather with colleagues from across the industry to discuss the state and future of MR. This year's keynote was particularly inspiring. Robin Pearl, vice president of Market Research at Estée Lauder, taking her lead from Tina Turner, asked, "What's Love Got To Do With It?" when it comes to achieving high-quality market research. Robin addressed the quality of sampling, questionnaire content and analysis.
Many issues are affecting sampling. Neither of the most popular quantitative methodologies - telephone interviews and Internet-panel interviews - now accurately represents the population. The 18- through 24-year-old demographic frequently uses only cell phones, putting them out of reach of most automated dialers, which only dial land lines. On the other side, the 55+ demographic is less likely to use the Internet. In addition, different ethnicities are harder to reach by different methods. As a result, Estée Lauder has encountered some of the following quality issues:
* Errors in Weighting - One survey reported in error that few women under the age of 25 used lipstick. This error was a result of the fact that the results had not been weighted to reflect demographics at all and had simply underrepresented that demographic. Robin also reported occasions where weights were applied to the final results rather than to the screener, providing results that did not accurately reflect that different demographic segments did in fact have different propensities of usage. When weights were applied, some research firms showed a willingness to weight a sample size of 50 up to 250 for a particular segment; only minor weighting should be applied, to prevent distorting the results.
* Oversurveying - Robin pined for the days when screeners typically excluded respondents who had participated in another survey within three to six months of the current survey. She joined a number of Internet panels and was shocked to soon be receiving 10 invitations a day. Professional respondents may in fact differ attitudinally from casual respondents.
* Panel Overlap - Estée Lauder is concerned about professional respondents being double counted. In one survey Robin sponsored, there was significant overlap of respondents across panels, to the point where through different panels the same household completed the same survey on the same day.
Besides issues with the quality of sampling, Robin has encountered quality issues relating to questionnaire design, especially for face, content and predictive validity. Too many surveys are fielded without adequate review of the survey instrument and without pre-testing. Pre-testing is important to determine which questions are unclear and what topics should be covered that aren't. One of Robin's pet peeves is questions that ask "What do you like about....?" rather than "What, if anything, do you like about...?" (I confess I'm guilty of this.)
Finally, Robin is concerned about issues with data analysis. She prefers top-box analysis to means, finding the reporting of means to be too reductive. If she had two products, each rated 3.0 on a 5-point scale, and one product was always rated 3 by respondents (0% on a 2-point top box), and one product was always rated either 1 or 5 (50% on a 2-point top box), she'd go with the "love it or hate it" product. She also pointed out that too often we as market researchers shortchange the conclusions and recommendations. Obviously, this work has to come last, requiring an accurate and detailed analysis to proceed it, but as a result we are often hitting against the final deadline and don't provide the insight and implications that we could have if we had budgeted more time.
Robin concluded her MRA keynote by saying that each of us need to have a Ph.D. when it comes to research. Not an academic degree, but "Passion, Heart and Drive". Quality research comes when you "love and respect your work", she said. That's what love's got to do with it.
Her keynote was well received by both end users and researchers alike in the audience. And while Robin was inspired by Tina Turner, I thought of another diva, Diana Ross: "You Can't Hurry Love." Many of the issues with quality come from our constant hurrying of the work. You can't hurry love - or quality.