Welcome to the Listening Post!

Data Quality & Validation Lessons from Panel Professionals

Tweet

 

Panel companies have opposite and competing forces that, if not carefully managed, can damage their businesses:

• Panels need to offer significant enough incentives to get people to take surveys regularly, yet too large incentives encourage respondents to cheat in a wide variety of ways, which the panel companies must then catch and remove.
• Long, tedious surveys produce low quality data and cause panelists to drop out, yet many panel customers are fielding just such surveys because those are the surveys for which they make the most money.
• To counteract high panel attrition, panels are often built and managed in a way that can't produce representative results. 

Financial incentives for survey results do create significant concerns about the quality of survey answers. As a result, panel providers carefully analyze respondent behavior. In our Lessons from Professional Panel Providers study, the most frequently cited concern by panel providers was for respondents speeding through surveys; 60% of the panels specifically check for respondents completing surveys much more rapidly than the norm. Other data checks are for "straightlining" (providing the same rating answer to many questions in a row), cited by 33% of panels; low quality or nonsensical answers to open-ended questions, cited by 23%; and "satisficing", going through the motions when answering questions, cited by 13%. As a result, 27% of the panel companies said that they manually review survey results. Another 37% check for duplicate accounts.

The percentage of panelists participating in any given survey they were invited to varies widely by length, incentive and targeted subsample. The panel companies provided ranges of participation, which averaged out to 27-55% across panels. Drop-out rates (the percent of panelists who abandon a survey before completing it) vary widely, ranging from 2% to 12%; such rates are a function of survey length and complexity.

Panelist satisfaction is particularly important to commercial panels, with 79% of panels doing some form of panel satisfaction. Postsurvey assessments are administered by 59% of panels. General satisfaction studies are performed monthly (2 panels), quarterly (4 panels), semiannually (1 panel) or annually (7 panels).

Panels face tension between low quality surveys and low quality responses. To address this, a few panels provide discounts to customers whose surveys are rated by panelists as significantly above average in satisfaction when compared to all surveys fielded to the panel.

What Panel Pros Can Teach Us about Data Quality & Validation

1. Conduct frequent panelist satisfaction research. Few in-house panels measure the satisfaction of their members. Better to use a monthly or quarterly rolling survey (sent to a different cohort within the panel each time period) rather than measuring satisfaction once a year as so many panels do.
2. Analyze implicit satisfaction measures. Panelist satisfaction surveys are an excellent explicit measure, but the implicit satisfaction that can be observed from panelist behavior is also important. Benchmark and track survey participation and drop-out rates against commercial panels.
3. Enforce best practices that lead to satisfied panelists. Don't let panel users field long and tedious questionnaires to panelists, but guide them through the process of creating short, engaging surveys.

Want to learn more? For the full results of this study, download our white paper, Panel Management Secrets: Lessons from the Professionals, or view our recorded research webinar of the same name.