Panel Effects: Practice & Conditioning

In his blog post "Practice Makes Perfect", Reg Baker, COO of Market Strategies, describes the winter Public Opinion Quarterly article, "National Surveys via RDD Telephone Interviewing Versus the Internet: Comparing Sample Representativeness and Response Quality" by Linchiat Chang and Jon A. Krosnick. Reg describes the actual experiment itself [I'd added explanatory links]:

As a broad summary...the online panel results showed less satisficing, less social desirability, better self-reports, and greater internal consistency than did the results from the RDD telephone sample. The online panel folks were just better at doing surveys.

As is typical of a paper by Krosnick, it contains an excellent summary of previous research on research. For our recent research webinar, "Customers as Confidants: Customer Panel Management Made Easy", I distilled this summary into the following high-level pros and cons of panelist participation in surveys.

Pros: Practice Effects

• Regularly answering surveys may enable respondents to improve the accuracy of their responses.
• Panel members may become more introspective and self-aware, improving their self-reporting.
• Respondents' answers to attitudinal questions also improve with practice.

Cons: Panel Conditioning

• The "stimulus hypothesis" that asking about future activity prompts that activity (e.g., asking participants if they will vote actual prompts greater turnout).
• Past surveys taken by panelists make those panelists less like the general population being researched.
• Panelist attrition nonrandomly affects panel representativeness, as members withdraw from panels based on the survey topics they've been subjected to in the past.

Overall, the research seems to indicate that the benefits of the practice effects outweigh the caveats of panel conditioning. Refer to the Chang and Krosnick article for full details.