Welcome to the Listening Post!

Net Promoter Score (NPS) Criticisms and Best Practices

Tweet

 

The Net Promoter Score®, popularized by Fred Reichheld in his book The Ultimate Question: Driving Good Profits and True Growth, is one of the simplest loyalty measures. Customers are asked "How likely is it that you would recommend us to a friend or colleague?" and then provide a rating from 0 ("Not at all likely") to 10 ("Extremely likely").

The measure is called the "net promoter" score, because detractors are subtracted from promoters, to provide the estimate of how many more promoters than detractors the organization has. Detractors are defined as respondents rating their likelihood to recommend 6 or less, with promoters only those who rated their likelihood a 9 or 10 (respondents who selected 7 or 8 are considered neutral). The NPS measure can run from -100% (0% promoters, 100% detractors) to 100% (100% promoters, 0% detractors), with typical measures in the 30-40% range.

Traditional customer-satisfaction measures typically omitted willingness to recommend, instead focusing on aspects like perceived value, customer satisfaction, corporate image, and rational and emotional commitment (see the ACSI model).

Avis, HP and IBM are among the many prominent adopters of NPS. The benchmark is popular for its simplicity, and Reichheld claims it correlates to company growth. Critics contend that it doesn't, that its 11-point scale has lower predictive validity than other scales, that the segmentation of promoters/neutrals/detractors is arbitrary and that other questions may be better predictors of growth rates:

• Not the Most Important Customer-Satisfaction Question - "We find no support for the claim that Net Promoter is the 'single most reliable indicator of a company's ability to grow' (Netpromoter.com 2006; Nicks 2006). Although we do not have access to the raw data from which these claims were made, we were able to compare some of the exemplar cases of Net Promoter with the ACSI, which Reichheld (2004) reports does not correlate with growth. Instead, we found that when making "apples-to-apples" comparisons, Net Promoter does not perform better than the ACSI for the data under investigation... The clear implication is that managers have adopted the Net Promoter metric for tracking growth on the basis of the belief that solid science underpins the findings and that it is superior to other metrics. However, our research suggests that such presumptions are erroneous. The consequences are the potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value, and shareholder wealth." - Timothy L. Keiningham, Bruce Cooil, Tor Wallin Andreassen, & Lerzan Aksoy, "A Longitudinal Examination of Net Promoter and Firm Revenue Growth"
• Doesn't Accurately Differentiate Promoters and Detractors - "The rule-of-thumb score classes proposed by Reichheld (promoters are those respondents who give a likelihood of recommendation of 9 or 10 while the detractors give 6 or less) are not supported statistically, mask important changes and potentially mislead management that there is negative NPS when this may not be the case." - Ken Roberts, Forethought Research Australia. Further, the standard NPS question itself is unipolar (willingness to recommend) but Reichheld's analysis treats it as bipolar (willing to detract vs. willingness to promote).
• Less Accurate than Composite Index of 3 Questions - "In his Harvard Business Review article ‘The One Number You Need to Grow', Reichheld maintained that since his tests showed propensity to recommend to be the single question that had the strongest statistical relationship to future company performance, there was no point asking any other questions in customer surveys... a single item question is much less reliable and more volatile than a composite index." - Customer Satisfaction - The customer experience through the customer's eyes, Nigel Hill, Greg Roche and Rachel Allen, p. 7
• Fails to Predict Loyalty Behaviors - "This research examines different customer satisfaction and loyalty metrics and tests their relationship to customer loyalty behaviors. The goal was to test the robustness of the customer-level analysis conducted by Reichheld and Satmetrix, which served as the foundation of their Net Promoter research. Contrary to Reichheld's assertions, the results indicate that recommend intention alone will not suffice as a single predictor of customers' future loyalty behaviors. Use of multiple indicators instead of a single predictor model performs significantly better in predicting customer recommendations and retention." - "The Value of Different Customer Satisfaction and Loyalty Metrics in Predicting Customer Retention, Recommendation, and Share-of-Wallet" (Timothy L. Keiningham, Bruce Cooil, Lerzan Aksoy, Tor W. Andreassen, Jay Weiner). NPS is attitudinal rather than behavioral, measuring how many people say they would be likely to recommend, rather than how many are doing so. A large body of research indicates that claimed intention is a better reflection of present attitudes than future behavior (Bird, Ehrenberg and Barnard).
• Performs Worse than Satisfaction & Liking Questions - The paper "Measuring Customer Satisfaction and Loyalty: Improving the ‘Net-Promoter' Score" by Daniel Schneider, Matt Berent, Randall Thomas and Jon Krosnick counterintuitively demonstrates that "satisfaction" and "liking" are better predictors of recommendations than "likelihood to recommend".
• Performs Worse than Other Scales - Schneider et al also demonstrate that the 11-point scale has the lowest predictive value of any of the scales tested. The authors recommend a 7-point scale with labeled ends and midpoint for the willingness-to-recommend question but also recommend a bipolar scale for a reworded variant.

As a result of this mounting criticism, only 19% of customer-feedback professionals agreed in a March 2008 survey that the NPS is a better predictor of growth than other loyalty questions and indices; 40% were neutral ("Customer Feedback Professionals Do Not Believe the NPS Claims", Hayes).

Despite these criticisms, NPS remains popular because it is well marketed, easy to understand and its model makes intuitive sense: every organization wants more promoters than detractors.

Some suggested best practices for using the Net Promoter Score in your organization:

• By all means, include the willingness-to-recommend question in your surveys.
• Use a customer loyalty index consisting of the willingness-to-recommend question with other questions relevant to your business (see Apostle Model Best Practices for one suggested index).
• Do not use the 11-point scale advocated by Reichheld, which is arbitrary in its assignment of promoters and detractors and has lower predictive validity than other scales. Reichheld shows flexibility as to the actual scale used; his original work showcases Enterprise Rent-A-Car, which uses a five-point scale in their research (treating 5 as promoters). Instead, use the seven-point bipolar scale recommended by Schneider, Berent, Thomas and Krosnick :

How likely is it that you would recommend us or recommend against us to a friend or colleague?

• Extremely likely to recommend against
• Moderately likely to recommend against
• Slightly likely to recommend against
• Neither likely to recommend nor recommend against
• Slightly likely to recommend
• Moderately likely to recommend
• Extremely likely to recommend

• As Reichheld suggests, use a follow-up open-ended question to probe why a respondent selected the choice that they did.
• Calculate the Net Promoter Score primarily for comparison to other firms, but use an arithmetic mean of your responses for internal tracking and benchmarking, as this will provide a more stable measure over time.

If you have time to read one white paper on the Net Promoter Score, read "Measuring Customer Satisfaction and Loyalty: Improving the ‘Net-Promoter' Score". Its authors include Jon Krosnick, the foremost researcher on questionnaire design, and Matthew Berent, Staff Survey Researcher with Intuit, which is one of the firms often cited for its use of NPS.

Update (9/09/09): Net Promoter Score is a Misnomer