Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
|
 RSS Feed
Posted by Jeffrey Henning on Thu, Nov 12, 2009
I'm adamantly against financial compensation for staff based on customer-service survey results. Invariably, it leads to customer-service agents gaming the system; this happens far more than executives are willing to acknowledge. For instance, a K-mart cashier posted a sign at the register saying, "For a chance to win a $2,500 gift card, log onto KmartFeedback.com and rate our customer service 9 or 10. Thanks for shopping at your 1 Penn Plaza Kmart!" Of course, people have a chance to win no matter what rating they give.
(photo: bcurrie)
A post on this example at The Consumerist led to a rash of comments giving examples for other major brands:
- CVS - "Yesterday I went to CVS for a prescription. When I checked out, the cashier looked at my receipt and stated that I had been selected to participate in a survey. Yeah, yeah, yeah, I know - this happens all the time, but she then showed me a mini candy bar and said, ‘you get a free Take5 candy bar so that will remind you to give us all 5's on the survey!' I thought it was pretty smart but I still didn't do the survey..."
- Enterprise - "I was asked to rate an Enterprise rental by the customer rep, on a scale of 1-10 (or whatever it was). I said 9 because I figure one should save 10 for over-the-top service; this particular rental went smoothly, but was nothing special. He kept after me, wanting to know what was wrong, why I didn't give them a 10, until I got the point that to them ‘10' means ‘normal service'."
- Home Depot - "The Home Depots around me [hand you a flyer] telling you the almost exact same thing."
- Nissan - "This reminds me of whenever I buy a car. The Nissan salesperson reminds me that I will receive a survey in the mail, and that I'm supposed to return that survey with all 5's (the best possible mark), or else their dealership will be dinged or knocked down some points or whatever. I don't generally return surveys anyway, and I certainly don't when someone tells me what my answers should be anyway."
- Old Navy - "An Old Navy store stapled a piece of paper to my receipt with a similar suggestion - something like ‘rate us a 10 and get 10% off'. Of course you get 10% off for filling out the survey no matter what, so I filled out the survey honestly, with perhaps a slight negative influence from the suggestion."
- Sears - "Did the same thing when I worked at Sears but it said all 10s and we were told to tell them all 10s or we'd fail. It would be nice if they used the surveys to better themselves."
- Target - "I have had plenty of cashiers at Target say that sort of thing too."
- Toyota - "A Toyota dealer actually told me to bring in the survey so we could 'fill it out together'. In exchange - he would fill my gas tank."
My recommendations:
- Don't financially incentivize any staff based on survey results. Doing so will change the results, no matter how strong your corporate culture is (for instance, Enterprise would discipline the representative mentioned above). Compensating only managers will lead to managers asking representatives to ask for higher scores. The most important thing is to gather authentic feedback about current satisfaction.
- Don't use survey results as a crutch for measuring employees. It's vital that you share rich, relevant feedback with agents so that they can serve customers better in the future. Their managers need to mentor them not overmeasure them. (See Employee-Customer Engagement Best Practices.)
- Don't use the Net Promoter Score as a transactional measure. From these and other anecdotes, it is clear that the NPS in customer-service settings leads to a collapse in the range and validity of the scale and an obsession with the top score, even for organizations like Enterprise that try to counterbalance this.
- Don't use numbers. They are arbitrary; in some pure mathematical realm they are better for averaging than the results of fully labeled scales but the lower interrater reliability of numeric scales negates that. What an 8 means on a 10-point scale varies from respondent to respondent. Use labels, as in these common rating scales, and make the highest labeled point hard to achieve: Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied.
Do you have any examples of being asked to provide artificially high ratings to surveys?
Posted by Jeffrey Henning on Tue, Oct 20, 2009
 Adding email triggers to a traditional questionnaire starts the transformation from a survey project to a survey process. This is especially important for transactional surveys. To add survey alerts to an existing survey, you will need to consider the following issues:
- Move from a random sample to an attempted census to maximize exposure to negative feedback. By replacing your random sample with an attempted census of everyone, you will have the opportunity to intervene with more individuals to attempt to improve their satisfaction. The trade-off to balance against this is that customers are now receiving surveys from you more often; this can be worth it, though, given the new opportunity to right previous customer-service wrongs. You are now going to catch more dissatisfied customers than before. Typically, for such transactional surveys, you will want to automate survey invitations, using input from an internal system such as a help-desk to drive invites. Implement touch-management rules so that people do not receive more than one survey of this type every 60 days.
- Adding or expanding the series of reminders. Emailing reminder invitations to potential respondents who haven't yet taken the survey can be an annoying nuisance. In fact, we often argue for the need to respect your potential respondent's time and interrupt them as little as possible. Reminders are an interruption, but in the case of survey alerts, you really want to make sure unhappy customers raise their hands so that you can help them. A series of reminders, each spaced 3 to 5 business days apart, helps achieve this.
- Develop the business rules for customer-service responses. Some organizations simply have survey alerts delivered to a single email address, often that of customer service (for customer surveys) or the internal help desk (for employee surveys). Others use multiple questions for triggers, and set up elaborate business logic to notify the most appropriate staff to issues they can assist customers with.
- Integrate with case management. Most organizations integrate email triggers into their existing case-management systems, so that survey alerts are prioritized and acted on alongside more traditional cases. Integration can be as basic as using the emails as inputs, to more sophisticated systems integration involving web APIs (Application Programming Interfaces).
You can certainly simply slap on a survey alert to an existing project, but we encourage you to plan a more detailed implementation of email triggers, to move from measuring customer satisfaction to intervening to improve it.
Posted by Jeffrey Henning on Thu, Aug 13, 2009
One of the exemplary aspects of the Domino’s Pizza transactional survey is that the questionnaire is updated in real time to display the name of your cook and the name of your delivery driver. This clearly communicates to the respondent that they are rating employees and that this feedback will most likely make its way back to those employees.  How valuable an approach is this? As Brian Koma reported in his summary of our Customer Experience study findings, one of the CEM best practices with the highest correlation to customer loyalty is sharing with employees the ratings of them done by customers. So expanding your existing transactional surveys to close the feedback loop in this way can help you improve customer loyalty over the long term. That said, it does require changing much about your survey process. Many transactional surveys are pulse surveys, taking the overall pulse of customer satisfaction with your service, rather than providing detailed diagnostics. Here are some of the changes you will need to make: - Larger Sample - A random sample of, say, 1 out of 5 customers is perfectly appropriate for the traditional pulse survey and provides an accurate measurement of overall service levels while not inviting every customer. Unfortunately, if you want to be able to provide cross-tabulations by employee, you will want at least 30 responses per employee. As a result, you will need to increase the incidence rate dramatically, possibly even moving to an attempted census, contacting every single customer.
- New Personalized Reports for Employees - Often, when moving to transactional surveys broken out by employee, the desire is for fresh data about an employee daily. That is often misleading, and we encourage weekly scorecards instead. One day’s results are simply too small to be meaningful. In fact, with small cell sizes per employee, it may even be necessary to provide a rolling multi-week average of an employee’s ratings rather than an average for just that week.
- New Personalized Reports for Managers – A daily summary of results for all of a manager’s direct reports, treated as a group, provides trend tracking that can act as an early warning system, alerting a manager to important issues. Hierarchical reports can even be rolled up to provide higher levels of aggregation, by contact center or division, for instance.
- New Executive Reports – Another useful way to analyze the data is by reviewing all the open-ended comments for the highest-rated employees and contrasting them with the comments for the lowest-rated employees. This can then be used to develop training materials to educate low performers about the practices that have proven successful with the high performers.
While the data gathered from an employee-segmented transactional survey is most valuable as a way of coaching and mentoring employees, for some staff it will be necessary to use this data to justify terminating them. With the employee-employer relationship being the most regulated relationship in America, you will want to make sure that you are collecting and analyzing this data in an objective and appropriate manner.
When first rolling out such a system, it is far better to concentrate on the opportunities for improving employee engagement with customers, providing valuable feedback to employees to help them serve the customer better. That’s the big opportunity to build customer loyalty.
Posted by Jeffrey Henning on Fri, Jun 26, 2009
 The follow-up or transactional survey concerns itself with getting customer feedback regarding a specific transaction, such as a purchase, a call to a contact center, a request for service or a product return. Such surveys can be conducted for multiple reasons. They are a great way to perform quality control to determine the level of service being provided and can be used to determine inconsistencies in providing service. Follow-up surveys can identify dissatisfied customers so that service recovery can be attempted and can measure the effectiveness of service staff. Here are some of the more common mistakes I’ve seen when organizations conduct transactional surveys: - Asking respondents to specify details about the transaction rather than using data integration behind the scenes to record that information
- Allowing service staff to select or influence potential respondents, for instance, by transferring some calls but not others to an IVR system—this results in skewed results that typically overstate satisfaction
- Failing to include any survey alerts or email triggers to enable service recovery to be attempted
- Failing to invite the recipient more than once to take the survey, which can result in bias
- Compensating employees on survey results in a way that encourages gaming the system and unethical behavior
- Not sharing survey results with service staff
- Having the questionnaire take more time to complete than the transaction itself—here’s a retail example and an auto club example
- Inviting participants on a monthly basis rather than weekly or daily—respondents are typically unable to answer in detail after more than a week has passed
- Conversely, inviting participants to take the survey before the incident is resolved
- Failing to implement touch-frequency rules where respondents are not invited too often; for instance, not invited more than once in a 30-day period
- Failing to implement a survey unsubscribe process so that customers can opt out of recieving surveys altogether
What mistakes have you seen in transactional surveys you've taken?
Posted by Jeffrey Henning on Fri, Jun 12, 2009
Since the rise of web surveys, we’ve talked about real-time feedback. Domino’s Pizza is carrying it to the next level. Working late last night, I ordered a pizza online for me and a co-worker, and it actually built an image of the pizza (shown in the corner) as I added toppings. But even the survey itself changed before my eyes, as the cook and delivery person were assigned and appropriate question wording was updated dynamically!
Where most restaurant sites give you a confirmation page, Domino’s gives you a Pizza Tracker: a dashboard for your pizza delivery.
The handy progress-bar/thermometer keeps you glued to the page:
But wait, I can’t answer the other questions yet, because they know the pizza hasn’t even been delivered. I get the validation message “Can you let us know after your order arrives?”
It’s a unique problem because transactional surveys are never delivered during the transaction. Normally transactional web surveys are sent a few hours or days later, with rules to screen out respondents who waited a week to reply, as they are assumed to no longer have a clear recollection of the transaction. Want the clearest possible recollection of the transaction? Ask them in the middle of it in an engaging fashion.
The actual question itself blew me away. “Christopher custom made your order. How did everything taste?” I can’t talk some of our customers into integrating their surveys with their CRM systems using standard APIs, and here Domino’s built a custom application that not only infuses data for Christopher in real time (behind the screen no doubt is an employee ID) but actually updates the text of the survey once they know who it is. (I now know the guy who made my pizza by name, something that hasn’t happened to me since high school.)
Asking the question with this wording was no doubt carefully thought out, but I’m of two minds about it. First, it’s clear that this rating reflects on Christopher (whose pizza making skills are excellent, by the way) and that my answer will have an impact on Christopher of Domino’s #3723 in some way. That’s good, and I like it. On the other hand, unlike the wording of the customer-experience assessment, this wording encourages ratings inflation: I really don’t want to give Christopher a low rating. I too well recall the joy of being a short order cook. (To the patron of McDonald’s #3570 in the spring of 1987 whose Quarter Pounder patty was absentmindedly cooked twice: I am so sorry.)
And just because the staff at Domino’s are showoffs, they do it again on the third question, updating it once they know that John is the delivery person:
“How did John your delivery expert do today?” dynamically replaces the previous and overly corporate wording of “Our goal is exceptional delivery. How was your delivery experience?” (For the record, I usually like my delivery experiences with epidurals and don’t-get-me-started on how Ridley Scott had to have been inspired to film Alien by an awful delivery experience.)
The last question, an open-ended question, is as friendly as they come, from “Use this handy box” to the request for “advice, grumblings or compliments”:
Finally, Domino’s does something pedestrian, limping across the finish line after an amazing race. The confirmation page asks me to call them if I need a response:
Call them? Can’t I text them or something? Certainly I should call if there is something wrong with the pizza and I want it fixed this minute, but this would be an appropriate place to ask for my email address if I wanted a response. They’ve already had me give them my physical address and phone number when placing the order. Still, a minor quibble to a phenomenal transactional survey.
Let’s recap:
- Domino’s took a customer-service question (“Where is my pizza?”) and transformed it into a hip, Web 2.0 opportunity to conduct a transactional survey.
- Domino’s made sure that its transactional survey didn’t detract from the actual customer experience but actually enhanced the experience.
- Domino’s has set themselves up for a large response rate. At a minimum, many people will rate two questions, and if they leave the web page open (thanks to the joy of tabbed browsers), then when they come back to their computer after eating their delicious pepperoni-half-onion-mushroom-pepper pizza, they will answer three more questions.
- Domino’s has tightly integrated their feedback platform with their operational systems, to the point where they make it clear to me by name the cook and delivery person that I am rating. (John, I should point out, was very friendly and seemed genuinely interested in what exactly Vovici does in its hallowed halls.)
- Domino’s demonstrated the utmost respect for respondents: the questionnaire was short and sweet and even conversational. Yet this is not a five-question survey, though respondents only have to answer five questions. Behind the scenes they integrate the feedback with the order itself and the operational data, collecting:
- The respondent’s phone number
- The address, allowing geographic analysis by city and state, and—by using the zip code—demographic analysis as well
- Whether this was an order sent to a home or business (specified with the address)
- All the attributes of the order itself
- The fact that they tried to upsell me the Buffalo Chicken Kickers® and I declined
- The time and date of the order
- The cook
- The delivery person
- …and some other fields I’m probably not thinking of
Domino’s, when it comes to transactional surveys, truly you are the upper crust.
Posted by Jeffrey Henning on Thu, Jun 04, 2009
The gap model (also known as the "5 gaps model") of service quality is an important customer-satisfaction framework. In " A conceptual model of service quality and its implications for future research" ( The Journal of Marketing, 1985), A. Parasuraman, VA Zeitham and LL Berry identify five major gaps that face organizations seeking to meet customer's expectations of the customer experience.
The five gaps that organizations should measure, manage and minimize:
- Gap 1 is the distance between what customers expect and what managers think they expect - Clearly survey research is a key way to narrow this gap.
- Gap 2 is between management perception and the actual specification of the customer experience - Managers need to make sure the organization is defining the level of service they believe is needed.
- Gap 3 is from the experience specification to the delivery of the experience - Managers need to audit the customer experience that their organization currently delivers in order to make sure it lives up to the spec.
- Gap 4 is the gap between the delivery of the customer experience and what is communicated to customers - All too often organizations exaggerate what will be provided to customers, or discuss the best case rather than the likely case, raising customer expectations and harming customer perceptions.
- Finally, Gap 5 is the gap between a customer's perception of the experience and the customer's expectation of the service - Customers' expectations have been shaped by word of mouth, their personal needs and their own past experiences. Routine transactional surveys after delivering the customer experience are important for an organization to measure customer perceptions of service.
Each gap in the customer experience can be closed through diligent attention from management. Survey software can be key to assisting management with this crucial task.
Posted by Jeffrey Henning on Fri, May 22, 2009
 As I mentioned last Friday, I attended a webinar, "The Ultimate Question: How to Measure & Build Customer Loyalty in the Support Center", presented by Fred Reichheld on the use of the Net Promoter Score® within customer support centers. Fred reviewed the thinking behind NPS and presented this questionnaire for use as a customer-service survey:
Considering only your most recent support experience, how likely would you be to recommend our customer support to a friend or colleague? (0 is least likely, 10 is most likely)
( ) 0 ( ) 1 ( ) 2 ( ) 3 ( ) 4 ( ) 5 ( ) 6 ( ) 7 ( ) 8 ( ) 9 ( ) 10
Please give your reasons for the rating above.
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Considering your complete experience with our company, how likely would you be to recommend our products to a friend or colleague? (0 is least likely, 10 is most likely)
( ) 0 ( ) 1 ( ) 2 ( ) 3 ( ) 4 ( ) 5 ( ) 6 ( ) 7 ( ) 8 ( ) 9 ( ) 10
Please give your reasons for the rating above.
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Fred described this as a bottom-up measure of NPS. He pointed out two sources of bias: the most recent transaction biases the respondent's thinking about the overall relationship, and the fact that the survey is sponsored by the firm gives scores an upward bias as respondents are less negative than they would be if the survey was anonymous.
In contrast to this, Fred described a top-down measure of NPS as a double-blind survey of customers of the sponsor and its competitors. This top-down average score will be different because of the biases inherent in the bottom-up method. Just as accounting reports can legitimately differ, these two measures, to Fred's mind, have valid, legitimate differences.
Given Fred's advocacy of short surveys, I was disappointed that he didn't point out that customer-support surveys really shine when additional data is integrated behind the scenes into the survey. This preserves the survey experience for respondents while enabling the survey analyst to study performance across many additional factors that affect customer loyalty, such as demographics and product ownership. This provides a much richer source of information for the organization to use to adapt and grow.
[Net Promoter Score is a registered trademark of Fred Reichheld, Bain & Company and Satmetrix.]
Posted by Jeffrey Henning on Fri, May 08, 2009
Though it was little noted at the time, in December the Corporate Executive Board introduced a new loyalty metric, the Customer Effort ScoreTM:
Exceeding customer expectations has long been the measure of success in customer service interactions-89% of customer service executives believe that "delighting the customer" will lead to increased loyalty. New research by the Corporate Executive Board's Customer Contact Council, however, reveals the alarming truth: exceeding customer expectations results in virtually no loyalty gains. In fact, service and support centers have little stake in building customer loyalty at all.
"The probability that a service interaction will drive disloyalty is approximately four times greater than the chance it will create any positive loyalty impression. In other words, as a function, customer service typically plays on the ‘negative side' of the loyalty field," said Matthew Dixon, Ph.D., Managing Director of the Customer Contact Council. "Most service executives are using traditional customer satisfaction (CSAT) or the more recently popularized Net Promoter® Score (NPS) to gauge loyalty in service interactions, but we found these metrics fail to capture the most powerful driver of disloyalty-the amount of personal effort a customer has to put into the service experience." In light of this new understanding, the Customer Contact Council has developed an original metric that is far more predictive of loyalty than either CSAT or NPS. This new metric, the Customer Effort ScoreTM (CESTM), is based on a single question that determines the degree of required customer effort during a service request.
Unfortunately, the actual wording of the single question is only available to Corporate Executive Board subscribers. The Customer Contact Council has published a presentation, "Shifting the Loyalty Curve: Mitigating Disloyalty by Reducing Customer Effort", which provides an introduction to the research. In a blog post, the author of the study provides additional detail about customer effort by service category and industry. The blog Every Experience Counts reports on the metric in a little more detail.
The selection of the CES metric was derived from a survey using a convenience sample of almost 18,000 customers of CEB clients. CES suffers from similar problems to NPS: it's proprietary and the data used for deriving it is not publicly available, making it difficult for third parties to verify the claims about the methodology. That said, I would encourage any contact center managers who are already CEB customers to check out the full report. It requires little effort (ahem) to integrate the Customer Effort Score into existing transactional surveys, which certainly makes it worth piloting.
Most surprising to me was the finding that 89% of customer-service executives look to customer satisfaction as the primary driver of customer loyalty. After all, it was over 13 years ago, Jones and Sasser published their landmark paper "Why Satisfied Customers Defect", introducing the Apostle Model and demonstrating that customer satisfaction and loyalty were orthogonal. Clearly the word is not yet out that satisfaction is but one component that drives loyalty.
Posted by Jeffrey Henning on Mon, May 04, 2009
Ironically, large organizations seeking to better research customer experience don't even understand their customer's experience with their research.
Let's look at the feedback process from the perspective of the typical customer of a large organization: the customer is surrounded on all sides by survey requests.
In this example, survey invitations come from the training department, the customer service department, the marketing department and the sales department. Each department has selected its own survey tool with no central coordination of how frequently customers are invited to take part in research. In fact, one Vovici client had subjected customers to 450 different survey initiatives in the prior year. The result?
- Customers stopped taking requests for surveys seriously and responded less often
- As response rates dropped, individuals invited even more customers to take surveys
- Survey content often overlapped with that of other surveys
- Significant time and energy was invested in gathering survey results, but those results were not even shared within departments, let alone between departments
- Information was gathered inconsistently by different authors, limiting:
- the ability to study changes in customer attitudes over time
- the ability to compare results across departments and product lines
- Wasted investment in redundant tools and surveys
- No one had a true, holistic picture of the customer experience.
Any organization serious about understanding its customer experience needs to start by analyzing and standardizing its own research into that experience. A feedback assessment is a great place to start on the path to enterprise feedback management.
Posted by Jeffrey Henning on Thu, Apr 16, 2009
 Yesterday, at the SCORE 2009 conference, I had the good fortune to hear Bruce Temkin of Forrester Research present "The Path to Customer Experience Excellence: Why, What and How". I regularly read his blog, Customer Experience Matters, so I was familiar with much of the material, but where the blog provides bite-size snack, Bruce's presentation was dinner and a show. (Complete with corny-enough jokes that I immediately reused one later in the day!)
Bruce began by walking the audience through a customer interaction that a Forrester analyst Adele had with a consumer-electronics retailer. His framing analogy was that too many organizations make customers roll a rock up the hill, like Sisyphus, only to have the rock roll back down. Adele used the site, but didn't get what she needed. The rock rolls back down the hill. Adele called customer service, but didn't get what she needed. The rock rolls back down the hill. Adele called technical support, but didn't get what she needed. The rock rolls back down the hill. Adele went back to the site and sent an email, but the responding email didn't give her what she needed. The rock rolls back down the hill.
As a guy who once named his company after a Greek demigod, I appreciated Bruce's take on this story. The two lessons he quoted:
- "Making customers push rocks up hills does not build loyalty."
- "Don't mess around with Zeus."
Here is my quick recap of the "Why, What and How" to customer experience excellence, with links to related blog posts from Bruce:
- Why? You could have safely ignored customer-experience management three years ago, but not now. Customer experience is growing up; it's been important to firms for long enough that your competitors are differentiating themselves on customer experience. The recession is strengthening the correlation between customer experience and customer loyalty, across all 12 B2C industries Forrester studied. Perhaps as a result, most businesses will cut other areas disproportionately more than they will cut areas that affect customer experience.
- What? One of the top firms in Forrester's customer experience rankings is CostCo, which tied for #3 after Barnes & Noble and USAA. "Great customer experience is not about Rainforest Café with a dazzling impression: it's about consistently meeting the needs and beating the expectations of customers. Unlike when going to a 7-11, you don't even expect to be able to park near the building at CostCo. But you do expect a broad selection at a great price." [quote from my notes; not verbatim]
- How? To build great customer experiences, your organization needs to follow the three principals of Experience-Based Differentiation:
- Obsess about customer needs, not product features.
- Reinforce brands with every interaction, not just communications.
- Treat customer experience as a competence, not a function.
Bruce wrapped up by pointing attendees to his free booklet, The 6 Laws of Customer Experience: The Fundamental Truths That Define How Organizations Treat Customers. Bruce is one of the leading authorities researching CE today; you'd have to have rocks in your head not to review his findings and work to make life easier for your customers.
Error sending email
Email sent successfully
|