Welcome to the Listening Post!

Mistake: Building Opt-In House Panels

A manufacturer launched a recruitment campaign to build an opt-in panel of customers. Unfortunately, they had very low recruitment rates. They simply were not able to get sufficient sample to segment customers by product and by year of purchase as desired.

Best Practice: Empanel Everyone

In the ideal world, every single customer is on your house panel. As online transactions grow to represent a greater proportion of many firms’ business, they provide opportunities to synchronize a panel with the firm’s CRM system, empaneling everyone. This provides a sufficiently large panel to segment customers and to sustain a series of surveys to random subsets of the panel.

Mistake: Failing to Coordinate Research

One company invited all 60,000 of its web site visitors to a survey and sent reminders a week later. Their analytic requirements didn’t need a census; a survey with 400 responses would have been sufficient. Right after the reminder went out, the Board of Directors had an urgent survey they wanted to send out. Everyone who got it had just been sent two survey emails.

Best Practice: Prepare an Editorial Calendar

Don’t treat ad hoc surveys as ad hoc, but prepare a monthly plan of segments being researched, leaving some room for last minute needs. Allocate sample appropriately across the studies.

Mistake: Enforcing Survey Quiet Periods

Setting up rules to not survey panelists within a set period of time (e.g., 60 days) seems like a common courtesy but it can lead to “holes” in the sampling frame. A prior ad hoc survey of Belgian customers led to Belgium being underrepresented in the next general study.

Best Practice: Random Sample Despite History

When representativity is desired, randomly draw sample regardless of past survey participation. Considering the bell curve, this will lead to a few recipients receiving many survey invites, and some who receive few. Where this is impractical, carefully analyze the pool of available respondents to identify groups that might be underrepresented based on past surveys of subsets.

Mistake: Fielding Bad Surveys

The most common mistake most organizations make is to assume anyone can write a questionnaire. Again and again, we review surveys that are so poorly written that they cannot answer the business questions they are meant to address. Not only do bad surveys collect bad data, but long surveys annoy customers and lead to low response rates in the future.

Best Practice: Respect the Respondent

Let anyone in the organization write a survey, but don’t let just anyone publish it. Set up a workflow process where an experienced survey author reviews the draft questionnaires for best practices. The expert author needs to be a champion of the respondent, stressing the need for a pleasant survey experience, free of jargon, confusing questions and huge grids.

Mistake: Trusting IT Systems’ Data Quality

CRM systems frequently suffer from many problems: missing data, outdated information and duplicate fields among them. In one case, showing respondents the information held on file about them actually led to greater dissatisfaction because the information was so inaccurate.

Best Practice: Investigate Data Quality

Despite the problems with CRM data quality, integrating such information allows you to ask shorter, more targeted surveys, while enabling you to do a rich analysis based on the CRM data. Don’t trust the CRM system—verify! Make sure you understand existing policies and efforts regarding data cleaning. For new studies, prompt the respondent to repeat information and then compare that against the CRM records to assess its quality.