Less Really is More When it Comes to Response Scales

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Two years ago, Ziggy Zubric, owner of Marketing Endeavors, and I engaged in a passionate debate about a survey we were collaborating on. Ziggy made the case for replacing our traditional five-point scales with something more innovative. I decided his approach wouldn't work for the task at hand, but his enthusiasm and unique point of view stuck with me. Recently, when writing a questionnaire, I recalled his advice and decided to invite him to write a guest post about his work.  - Jeffrey

Every researcher (and every client!) should be interested in developing surveys that:

1. Look quick and easy to take
2. Generate maximal variance (i.e., good distribution among all response options)
3. Avoid "response set" (i.e., respondents mindlessly giving the same answer for every question)

After of years of conducting online surveys, we've developed some strong opinions on how best to do this, and it starts with (a) replacing traditional five-point scale items with three-point scales, then (b) using a non-traditional answer order.

For context, let me state that we focus heavily on the in-store customer experience. That is, after customers have visited a bank, restaurant, grocery store, retail outlet, etc., we want to discover their thoughts and feelings about the experience. 

Importantly, we do not believe in asking customers about specific details. For instance, unless a client demands it, we never ask customers to recall whether eye contact was made, if employees were wearing name badges, etc. Customers rarely retain accurate memories of such issues (why should they?), and if asked such questions, they simply guess.  

Thus, the most you can reliably do in a post-experience customer survey is to explore overall perceptions. Customers can't accurately tell you if a teller introduced herself, smiled, and expressed appreciation for their business, but they can report if the teller created an impression of professionalism, competence, compassion, patience, etc. (They can also accurately share their feelings on satisfaction, loyalty, brand perceptions, etc.) 

Benefit #1- It looks easier to take

First, our experience shows that respondents are far more willing to take and finish surveys that are visually simple and appear to require little effort. This leads to less fatigue and far greater completion rates. 

And let's face it, using only three answer options takes up a lot less visual space and is far less daunting than five options.
Thus, we always try to limit the response options to three, as that number offers the robustness to capture what we need while still remaining visually inviting.

Benefit #2- It generates more variance

One of the first objections people usually raise is that limiting respondent options to only three answers greatly reduces variance. But our experience shows that exactly the opposite occurs. 

Years of data show that the spread of a five-point scale usually skews heavily towards the top two answer options. Thus, when using five-point scales, we'll find that:

• 88% of respondents answer 4 or 5
• 10% answer 3
• 2% answer 1 or 2

We see this pattern whether using numbers (e.g., 1 through 5) or any variation of verbal descriptors (e.g., Very good, Above average, Average, Below average, Poor). 

The bottom line is that variance can't get much worse than it already is. 

This can't be valid

How believable is it that 88% of all customer experiences are ‘very good' or ‘excellent'? Does that jive with your experience? I've asked a lot of people that question, and not one of them has ever said "Yes."

The reality is that service experience data collected with five-point scales greatly exaggerates the quality of service. And practically, telling customers that their average score on a five-point scale is between 4.6 and 4.8 for every item simply isn't helpful. 

What clients ultimately want to know is: When are we so good/bad that the experience is memorable and noteworthy, and how often are we extremely good/bad?

According to five-point scale data, 88% of service experiences are Good or Excellent. But we all know that's not true. The majority of service experiences are average. 

But occasionally customer service is extremely wonderful or remarkably horrid, and those are the experiences we want to make sure we capture.

With five-point scales, a typical, efficient, mildly positive experience will be rated as a 4 by some respondents, and a 5 by others. And a negative experience will be rated as a 4 by some respondents, and a 3 by others. In effect, our categories stop being mutually exclusive, vitiating the validity of the data.  Ultimately, what we want to accomplish is to differentiate truly positive/negative experiences from the vast majority of average, just-fine experiences, and a five-point scale just doesn't get us there.

So why do respondents provide such glowing reviews? 

That requires a lot more discussion, but we're dealing with at least three psychological tendencies. 

First, respondents feel a sense of guilt about giving scores that are very low, so they avoid the bottom categories.

Next, respondents are far too willing to describe an average, typical, yet perfectly acceptable service experience as Excellent or Very good. 

Finally, respondents who are simply trying to get through the survey (aka, response setters) will mindlessly click the top answer option every time, creating higher positive scores. 

The three-pointer

Let's deal with overcoming the first two psychological tendencies. Basically, they both result in respondents being far too generous in their assessments. When we employ a three-point scale, we can word the response options to help customers provide more accurate feedback. 

Thus, our most negative option needs to sound very mild, making it less guilt-inducing. Our middle option needs to sound very positive, making it more acceptable as an option. Finally, our positive option needs to sound so stupendously, impossibly, incredibly tremendous that respondents understand to reserve it only for truly excellent experiences. 

Some wording examples we've used when assessing customer service:

• Above and beyond
• Very good
• Could use some tweaks

• Truly outstanding
• Perfectly acceptable
• Not the best

Clients that we have migrated to this system show more variance in their data, and the response options overlap far less, which increases validity. Now, the strong majority of responses fall into the middle category, which is just what we want.

And when someone checks "Truly outstanding," we are far more confident that our client's staff genuinely did provide extraordinary care. Similarly, when we see a "Not the best," we know that something was truly amiss. 

Also-and this might make traditionalists uncomfortable-don't be afraid to play with the wording. Different clients and different contexts require nuances. As long as you're consistent within that client (or within that industry if you're providing benchmark data), you'll be fine. 

Not to mention the fact that respondents are sick to death of traditional wording options, and they find unexpected answer options refreshing.

Benefit #3- Avoid response set

Sometimes respondents will put their brain on cruise control while taking a survey, and simply click on the same option (often the top one) for every question. To overcome this, list the "middle" option first. For instance, try:

• Very good
• Could use some tweaks
• Truly outstanding

Does this violate traditional ordering of categories? Yes. But so what? 

By listing your middle, unremarkable option first, people just mindlessly clicking the top option won't give you a false sense of good or bad news; and the respondents who truly have an extremely good or bad experience to relate will look for their appropriate response and check it, providing us more certainty that these really were exceptional experiences (for better or worse).

So overall ...

Vovici Online Survey Software

- Ziggy Zubric, owner, Marketing Endeavors