When Surveys Go Bad
Posted by Jeffrey Henning on Tue, Sep 09, 2008
Jim Davies, an analyst with Gartner, gave a great presentation today in National Harbor, MD, at the Gartner CRM conference, entitled EFM: The Who, When, Why, Where & What of Surveying. He started with four great anecdotes about surveys gone awry.
Lengthy Surveys - One day he had car trouble, and his auto club towed him home. The firm made the best of a bad situation for him. He was very pleased with the service, until the driver handed him a PDA and asked him to fill out a survey. He was happy to, but then the survey kept going, and going, and going: 43 questions in total. The survey itself soured him on his overall experience with the auto club! A customer satisfaction survey that led to dissatisfaction.
Impersonal Surveys - One day boarding an airline, a steward came to his seat, addressed Jim by name and asked him to complete a paper survey. He said he would be happy to, but then he looked at the survey. It asked his name, frequent flier number, what flight he was on, his destination, and so forth - information that the airline knew, but the survey firm didn't. This made a survey seem as tedious as filling out a form.
Inappropriate Surveys - A hospital sent a household a survey asking the patient to rate his recent operation. Tragically, the patient had died in surgery. His widow was not amused to receive the survey. In fact, she talked to a newspaper about the callous way she was treated, creating a public-relations problem for the hospital.
Disconnected Surveys - Jim said that even Gartner didn't always practice what it preached. The firm frequently ran multiple, disconnected survey initiatives, with analysts often unaware of similar surveys being fielded.
This was just one slide of the dozens Jim presented, and he didn't have time to go into detail on how each of these issues can be addressed. Here are some suggestions.
- Shorten Lengthy Surveys - You can address overly long surveys in three ways: relentlessly prune questions (an approach that will prove unpopular with your coworkers); break a long questionnaire into multiple studies, each fielded separately; or randomly show each respondent a short subset of a long questionnaire. If you are going to have a sufficiently large sample size, then you lose little statistical significance by breaking the survey up into bite-sized chunks. Remember, the mathematics of survey research are such that you do not need to ask every single member of your population their opinion; just a random, representative sample will do. The auto club should have chosen three questions they wanted to ask every customer, then had the PDA randomly choose from the forty other questions; if certain questions only made sense together, they could be set up as a block of questions always chosen as a unit. This way, the auto club would have collected all the data it needed, without making its customers wish they were still on the side of the road.
- Personalize Surveys - Many researchers are used to the old days of paper or telephone surveys, where if information was desired, it had to be asked of the respondents directly. With web surveys, and with web services that can connect disparate applications, surveys can now integrate information from customer relationship management systems, HR systems and other key databases. Yes, this takes more work for the airline to set up the integration, but it would enable the customer to spend their time answering the attitudinal questions in detail, rather than wanting to avoid filling out the form by jumping out the window and using their seat as a flotation device.
- Keep Good Lists of Respondents - Every good survey process should have a global unsubscribe database built in. And the business should have processes set up to add entries to that database. Fortunately, few of us are in industries where our customers may die in our care, but - for those who are - it's all the more important to include that risk in their survey-research processes.
- Connect the Surveys - Most likely your organization has empowered you and your coworkers to make the investments you need to be successful. And often that involves using your credit card to purchase a survey. As an alternative, enterprise feedback management systems empower users to create the surveys they need to field in order to be successful in their jobs, while providing some centralized control to make sure customers are treated with similar respect, and not oversurveyed. Deploying an EFM system across your organization can be a daunting project, but if you start with one brand or one division, you can begin to treat your customers with more respect and better leverage the insights you're gaining from them, and then roll out the solution to other divisions over time.
When surveys go bad, make sure to reform them!