ARF Online Research Quality Council
Posted by Jeffrey Henning on Wed, Oct 28, 2009
Joel Rubinson, the Chief Research Officer of the ARF, discussed ARF quality initiatives. The ARF ORQC (Online Research Quality Council) was assembled to address the issue of panel quality. The committees developed "Foundations of Quality", a research-on-research project in October and November 2008 conducting 100,000 surveys across 17 U.S. panels, supplemented by phone and mail research.
The research revealed that all seventeen panels could retest their results reliably, but results varied significantly from panel to panel. Switching panels or using a panel undergoing major changes could change the results. The purchase intent for a concept correlates to panelist longevity (panels vary significantly by longevity), but no weighting scheme removed this variation when doing multi-panel sourcing. Rebutting assumptions, taking multiple surveys per month (3 to 10) was actually good for respondent engagement, putting the professional in the term professional respondent, which is usually a disparaging terms. Most panelists were not in it for the money, but those who were provided lower quality answers.
An industry-solutions committee started translating the insights from this research into an action plan, the QeP. The QeP (Quality Enhancement Process) v 1.0 offers templates, definitions, metrics and declarations to bring structure to conversations between buyers and sellers about online panel quality. The QeP is designed as a process to help buyers and sellers meet the shared goal of providing valid and consistent data. QeP is designed to encourage innovation that improves data validity across the online research ecosystem, which includes marketing and research departments on the client side, management consultants, suppliers and subcontractors to those suppliers.
The QeP looks at standards of quality at the panel level, sample/study level and the survey/response-quality level.
- At the panel level, panel providers should document recruitment methods, incentive systems, panelist profile, privacy policy, survey QA standards, etc. When panels merge, buyers should be wary of doing trend analysis with results from a predecessor panel.
- At the sample/study level, document a consistent sampling plan, promote sample consistency and eliminate duplicate survey taking in any form (deduping households, not just respondents).
- At the survey/response-quality level, control for panelist longevity.
The next step for ARF is to pilot test the QeF to ensure that the templates are clear, the documents are workable and the reports are useful and comprehensive. These results will drive future evolution of the QeP.
Update - Joel Rubinson uploaded his presentation to SlideSlide: