Survey Software, Web Survey, Online Surveys, and Enterprise Feedback Management solutions from Vovici

Your email:
   

Welcome to the Listening Post!

Your single source for everything Voice of the Customer (VoC) and Customer Experience (CxP). And, don’t forget you can follow us on twitter @vovici, or come check us out on Facebook and join the Vovici Network on LinkedIn.

 

Current Articles | RSS Feed RSS Feed

Rating Scale Comparison: Weighing Different Scales for Survey Research

 

Rating Scale Comparison - Weighing Different Scales for Survey ResearchIf there is one thing I dislike, it is giving people advice about scales. As Brian Koma, our vice president of marketing puts it, "People will defend which scale they use like they're fighting a holy war."

So here's some advice on why the three-point scale, the five-point scale, the seven-point scale, the ten-point scale and the eleven-point scale are all the absolute best rating scale for you to use!

  • Three-item scale: "Using only three answer options takes up a lot less visual space and is far less daunting than five options.  Thus, we always try to limit the response options to three, as that number offers the robustness to capture what we need while still remaining visually inviting." - Ziggy Zubric, owner of Marketing Endeavors, "Less Really is More When it Comes to Response Scales"
  • Five-point scale & seven-point scale: "To explore the relation between scale length and reliability, we conducted a meta-analysis of the results of many past studies. Our data consist of results from 706 tests of reliability taken from thirty different between-subject studies. We combined various measures of reliability and various sample sizes, controlling for these and other factors in determining the relation of scale length to reliability. In general, we found that five- or seven-point scales produced the most reliable results. Bipolar scales performed best with seven points, whereas unipolar scales performed best with five." - Jon Krosnick, professor of communication at Stanford, "The Optimal Length of Rating Scales to Maximize Reliability and Validity"
  • Ten-point scale: "A five-point scale is totally inappropriate for customer satisfaction studies. Why? It lacks enough granularity and robs companies of a burning desire to take corrective action. It commonly leads executives to believe that ‘80% rate us four or five; that's great, let's move on,' without realizing that it simply means that 80% are at least somewhat satisfied. Further, many people will never rate anything a ‘five,' resulting in ‘four' including those who are really very satisfied and those who are only somewhat satisfied. To avoid this topping effect, use at least a 10-point scale and count nine and 10 ratings as fully satisfied. This will also allow easier analysis of what bottom-line effects satisfaction has, since such tools as regressions work better with a more granular score." - Brad Bortner, principal analyst with Forrester Research, "Best Practices: Why Customer Satisfaction Studies Fail"
  • Eleven-point scale: "The 0-to-10 scale has many significant advantages: Customers find that the scale makes intuitive sense..., most of the world already uses the metric system..., customers may refuse to give anybody a perfect score..., customers will transpose the top and bottom on a 1-to-10 scale..., scales with fewer points seem more susceptible to grade inflation..., the 0-to-10 standard is being adopted by many of the world's leading companies." - Fred Reichheld, Fellow with Bain & Company, The Ultimate Question, p. 98-99

What's your favorite size for a rating scale, and why do you prefer it?

Comments

Appropriate scale length depends on the extent respondents can differentiate different attribute levels in the stimuli they are rating. So: for concepts tests in which respondents are being exposed to something new, a 5- or 7-point scale will be best. For a customer satisfaction study, where respondents are rating products with which they are familiar, then an 11-point scale (0 to 10) is best. In all cases, use a balanced scale. If you go to the “bible” of psychometric theory—Jum Nunnally’s Psychometric Theory—scales measure the amount of attributes. Consequently, they have to have a midpoint.
Posted @ Wednesday, June 17, 2009 8:59 AM by Terry Grapentine
I am not the greatest writer but I do alot of Marketing research. I recently had to convince "management" about switching from a 5 to 7 point scale. I found an simple article from Quirks and though it would be Relevent to this discussion. 
 
basically there is no real conclusive evidence to suggest a one scale system. the question know becomes unipolar or bipolar- It seems Maritz uses the unipolar more 
 
 
 
Balanced scales have equal numbers of positive and negative points: completely satisfied; mostly satisfied; mostly dissatisfied; completely dissatisfied. Although there are exceptions, a best practice with respect to balanced scales is that the use of modifiers should be symmetrical on the positive and negative ends of the scale. 
 
 
 
Unbalanced scales attempt to get greater discrimination on one side of the scale than on the other. If past experience suggested that most respondents are satisfied with a certain product or service, a researcher might want to “stretch out” the positive side of the scale, as in this extreme example (five positives and only two negatives): completely satisfied; very satisfied; mostly satisfied; somewhat satisfied; barely satisfied; mostly dissatisfied; completely dissatisfied. 
 
 
 
Whether a scale should be balanced or unbalanced usually depends on whether we’re measuring a unipolar or a bipolar concept of satisfaction. A unipolar satisfaction scale might range from “not satisfied” to “completely satisfied” (i.e., it doesn’t measure any more extreme dissatisfaction at all). In contrast, a bipolar scale would range from “completely dissatisfied” to “completely satisfied” (i.e., it measures extremes of both satisfaction and dissatisfaction). 
 
 
 
When we don’t know ahead of time whether most respondents will tend to be satisfied or dissatisfied, or if we expect high levels of dissatisfaction, a balanced bipolar scale is appropriate. If we know from past experience to expect low levels of dissatisfaction, an unbalanced unipolar scale will be better. In our firm’s experience, most studies show low levels of dissatisfaction, so our most frequent scale recommendation is for an unbalanced satisfaction scale. 
 
 
 
Some people think that satisfaction and dissatisfaction should be measured separately and are different entities. Although Maritz has observed that satisfaction and dissatisfaction may sometimes have different drivers, and there may be nonlinear relationships between satisfaction and other variables, we have seen no compelling reason to measure them separately. 
 
 
 
C. Midpoints 
 
 
 
A midpoint communicates neutrality on a balanced scale. “Neither satisfied nor dissatisfied” serves as the midpoint in this balanced bipolar scale: completely satisfied; mostly satisfied; neither satisfied nor dissatisfied; mostly dissatisfied; completely dissatisfied. 
 
 
 
There seems to be no difference in quality between scales that have a midpoint and those that do not. For bipolar scales, however, Maritz Research follows the advice of Sudman and Bradburn (1982) to “include the middle category unless there are persuasive reasons not to do so.” Krosnick (in press) suggests the same. 
 
 
 
D. Number of scale points 
 
 
 
Despite the strong opinions of some scale enthusiasts, there just isn’t powerful empirical evidence that a single number of scale points are always the best. A literature review by Cox (1980) summarizes the consensus regarding self-administered surveys: “Seven, plus or minus two, appears to be a reasonable range for the optimal number of response alternatives.” Internal research has not indicated a consistent difference in quality between 5- and 10-point scales. Krosnick (in press) finds that 5-point scales have the greatest test-retest reliability in paper-and-pencil surveys. Maritz’ experience with mail studies has shown that 5-point scales provide better dispersion of responses (and are, therefore, more discriminating) than other scales. We expect these results would generalize to other visual survey modes (e.g., PC or Web-based). 
 
 
 
The findings about the superiority of 5-point scales apply specifically to visually perceived scales and may or may not generalize to questions that respondents hear rather than see. Ten-point response scales are common in telephone surveys and have also been found to provide some advantages: greater statistical precision, statistical power, and room for improvement (Wittink and Bayer, 1994). We have also seen evidence that respondents from some ethnic groups shy away from using the extreme endpoints and may answer more accurately if they have a greater number of scale points from which to choose. Hence, while our basic recommendation is to use 5-point scales, we are comfortable using 10-point scales for some surveys, particularly those conducted via telephone. 
 
 
 
Posted @ Wednesday, July 08, 2009 3:29 PM by Stephen
For a seven point bi-polar scale is there consistent way to label the points?  
 
 
 
Extremely Satisfied 
 
Satisfied 
 
Somewhat Satisfied 
 
Neutral 
 
Somewhat Dissatisfied 
 
Dissatisfied 
 
Extremely Dissatisfied
Posted @ Monday, August 17, 2009 9:54 AM by Bob Klaus
That certainly works, though I prefer to start survey scales with the negative choice first. The 7-point satisfaction scale I use: 
- Completely dissatisfied 
- Mostly dissatisfied 
- Somewhat dissatisfied 
- Neither satisfied nor dissatisfied 
- Somewhat satisfied 
- Mostly satisfied 
-Completely satisfied 
You can label the midpoint “Neutral” for succinctness, if you prefer. 
Posted @ Monday, August 17, 2009 10:07 AM by Jeffrey Henning
Question: I am a graduate student conducting a survey to determine worksite safety culture from a construction workers perspective. I intend to use scaled questions in my survey. Example questions are the following: 
 
 
 
Please rate your level of agreement to the following statement: 
 
1. My coworkers support the company's safety program 
 
2. Employees correct other employees unsafe acts 
 
 
 
I considered doing a 5 point scale to measure agreement, but then considered using a 4 point balanced scale. My thinking is that either workers support the safety program or they don't. There is no "neither." As my sister would say, "Either you are pregnant, or you are not." 
 
 
 
I would appreciate any advice regarding this.
Posted @ Wednesday, October 07, 2009 12:29 AM by Jared Runge
Typically agreement scales are bipolar (disagree to agree), and the best bipolar scales have seven items. I'd use: Completely disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Completely agree 
 
Many respondents select "Neither agree nor disagree" (sometimes "Neutral" is substituted) as a "No opinion" measure.
Posted @ Friday, October 09, 2009 4:22 PM by Jeffrey Henning
On the whole, we use 10-point scales, labeling both end-points, with 1 = lowest measure (i.e., Does not describe me at all; Not at all important; poor, etc.) 5 - 6 labeled as the neutral measure (i.e., Describes me somewhat; somewhat important; average/acceptable, etc.) and 10= highest measure (Describes me completely; extremely important; excellent, etc.).  
 
When interpreting results, we typically look at means as well as Top-4 Box scores. We have found that unless we define the midpoints, we risk mis-interpreting the 7 - 10 scores, where some participants are using a grading system model when selecting their score and "7" is close to "failing" vs. the low end of extremely positive. 
Posted @ Wednesday, December 02, 2009 10:50 AM by Janet Metzelaar, President, Beyond Data Inc.
The 7+/-2 cognitive limit hides the fact that cognition is logarithmic. three is the baseline. At 5, you've selected or filtered another population. At 12, your population has gotten very small. Some people will not see the difficulties at 12, but the folks you lose moving from 3 to 4 will be more than 4x away from 12.  
 
Keep in mind that a logarithmic scale of base 3 puts item 4 at the linear location of 6, item 5 at 9. Twelve is way the hell out there for the 3 crowd.  
 
This has nothing to do with discrimination of the results. It's more like blowing away your statistical significance.  
 
Each cognitive population would need to be treated as its own entity, rather than as a contributor to the aggregate population. Aggregates falsify.
Posted @ Monday, December 07, 2009 12:51 PM by David Locke
No one has mentioned that agree-disagree items are prone to acquiescence bias (though Jeffrey hints at this).  
 
Notice also that in the original blog post, Krosnick's response is the only one citing scientific evidence for his recommendation.  
 
Read the literature.
Posted @ Friday, June 11, 2010 9:43 AM by Matt
You are right, Matt – we should read the literature. Unfortunately, for many researchers, rating scale best practices are a matter of faith rather than science, a matter of tradition rather than best measurement.  
 
And I would encourage people to rewrite agreement scale questions; that’s another tradition that should be retired.
Posted @ Sunday, June 13, 2010 9:39 AM by Jeffrey Henning
I appreciated your article because it provides info about when a scale is useful rather forcing the point of which is best. 
 
Once one has decided on the number of points, the next question is how label the points. From my work with my mentor Kerm Carlson (M.A in media research) who cited Likert's work, the 5 point scale provides more value distinction than a 3 point and is more intuitive for quick emotional accuracy than a 7 or 10 point. He avoided naming the points because it moved the respondent from intuitive/gut to intellect/analytical. He allowed for mid-point responses too, which gave respondents the option to express the accuracy they want (more democratic and therefore respectful). 
 
For example...evaluating "How do you feel about banana in a peanut butter sandwich?"  
 
Repelled Attracted 
1--2--3--4--5 
 
vs 
 
1 extremely repelled 
2 repelled 
3 neither repelled nor attracted 
4 attracted 
5 extremely attracted 
 
I suggest that the detailed words are unnecessary, distracting and at worse misleading when not congruent with the key criterion in the question. However, an indicator of the high and low end is essential. I use 1 for low and 5 for high so it is easy to do math and charting with the results.  
 
Underlying all this are the questions...Does my survey measure what I really want to know? and Will the answers provide info I can take action on? In my experience, many surveys fail this test. 
 
Rhetorical Survey: 
To what degree were my comments helpful? 
Unhelpful Helpful 
1--2--3--4--5 
Posted @ Saturday, February 19, 2011 1:35 PM by Doherty
Post Comment
Name
 *
Email
 *
Website (optional)
Comment
 *

Allowed tags: <a> link, <b> bold, <i> italics

Latest Posts

Loading
What's New
Independent Research Report: Why Customer Experience? Why Now?
VoC on Twitter
Verint Blog
Verint Blog: Read the Latest from the Verint Systems Blog