Standardization of Scales in Survey Analysis

While I believe that the scales with the highest reliability and validity are 5-point unipolar scales and 7-point bipolar scales, and I prefer using fully labeled scales, that is for gathering information from respondents. Presenting information to users of your survey research is a different matter. Simply presenting scales designed for accurate data collection may not facilitate ease of understanding of the results.

For instance, since 7-point scales are rare in business use, readers of your report may have difficulty understanding the results if you use a 7-point scale. If you are using a mix of 5- and 7-point scales when reporting results, you are certain to confuse some of your readers.  If you are doing a cross-survey analysis where similar questions in different surveys used different scales (see Standardize Your Customer Satisfaction Questions & Rating Scales), you will definitely want to standardize your scales.

For presenting data, I typically prefer to map scales to a 0-10 scale. I find that the business professionals I present to intuitively understand this scale. The broader range of values also makes it easier for readers to see differences in the results.

For instance, I would typically not present the following, even though this is an accurate representation of the gathered results.

Rating of Attributes on a 1-5 Scale
from 1 = Not at all important to 5 = Extremely important

Functionality of product	4.7
Product learning curve	4.6
Quality of technical support	4.4
Ability to grow with product line	3.9
Price of product	3.6
Availability of free trial	3.5
Helpful sales representative	3.4
Third-party reviews of product	2.9
Customer list of vendor	2.7
Vendor’s brand name	2.5

On a 1-5 scale, the midpoint is 3.0, not 2.5 (which would be the midpoint of a 0-to-5 scale). To calculate the midpoint, simply average the lowest and highest ratings: for instance, (1+5)/2 = 3.

Instead of presenting the above, I standardize the results to a 0- to 10-point scale. Standardization from a 5-point scale is not simply a matter of doubling the answers; doing that would produce ratings from 2 to 10. The 11-point scale has more than twice the granularity of a 5-point scale.  Instead, map the scale to a 0-to-1 scale: (X-1)/4 produces a number from 0.0 to 1.0 for each rating.  Then map this to a 0-10 scale, so (X-1)/4*10 results in the new scores.

Rating of Attributes on a 0-10 Scale
from 0 = Not at all important to 10 = Extremely important 

	0-10 Scale
Functionality of product	9.2
Product learning curve	9.0
Quality of technical support	8.4
Ability to grow with product line	7.3
Price of product	6.6
Availability of free trial	6.3
Helpful sales representative	6.0
Third-party reviews of product	4.7
Customer list of vendor	4.1
Vendor’s brand name	3.8

Standardizing bipolar scales typically involves mapping them to a –X through +X range. For instance, from -3 through +3 for a seven-point scale to -10 through +10.

Remember, your job as the survey analyst is not to provide an in-depth, detailed view of all the data collected, as it was collected.  Giving someone a spreadsheet of all the survey responses would do that. Your job as a survey analyst is to tease out the most important information and present to it your readers in an fashion that will maximize comprehension and understanding. And one tool to accomplish that is standardizing scales for presentation purposes.

Hierarchical Reports: So Survey Report Generation Isn't Like "Groundhog Day"

One retailer takes its core values very seriously, and its most popular survey is a monthly assessment asking each employee to evaluate their local store for compliance with those core values.  The climate survey asks questions about trust in employees, fairness at work, care and concern for customers, enthusiasm and day-to-day satisfaction.   With about 900 branches, 100 districts and 10 regions, the same survey is viewed over a thousand different ways.  Prior to automating this, administrative staff had to painstakingly prepare a thousand reports each month.  Talk about doing the same thing over and over again, a la Groundhog Day!
This corporate-climate survey is closely watched by all in management, and their needs are a great illustration of the capabilities of Vovici Feedback Intelligence:

• The branch manager and assistant branch manager can drill down on the survey data for their retail location in detail.  They can also compare their branch to overall ratings for their district, their region and the nation.  They can't view any other branch's ratings, though.
• The district manager can compare each branch in their district and can dive into the details for each branch, looking for what one branch management team is doing better than another.  They can compare their district to overall ratings for their region and the nation, but they can't view any other district's ratings and they can't review ratings of branches outside their district.
• The regional vice president and assistant regional vice president can compare districts and branches, looking for best practices that can be applied across the region.  They can compare their region to other regions and the nation, but can't view results for other regions, districts or branches.
• Corporate executives get dashboard reports showing high-level trends and comparison of regions and can drill down in detail to the results for any branch, district or region.

Vovici Feedback Intelligence excels at managing the thousands of users and report views that help bring enterprise-wide surveys like this one to life.  While implementing Feedback Intelligence won't win you Andie MacDowell, we can guarantee that you will win the love of your staff who are manually compiling these reports day after day after day.

Survey Analysis & Business Intelligence

Last week we announced the latest product in our enterprise feedback management suite:  Vovici Feedback Intelligence.  Most EFM and survey software applications take a traditional view of survey analysis, providing frequent reports, verbatim lists and cross-tabulations.  While a survey tool can report on each question intelligently, using a pie chart for a choose-one question, a bar chart for a choose-many question, and so on, most survey tools are not meant for ad-hoc analysis.  Such tools often make it difficult to manipulate and share survey data.  In fact, many of our prospects were exporting survey data to Excel, SPSS or SAS.

The genesis of, and the genius of, Vovici Feedback Intelligence is that it for the first time marries the flexibility of Business Intelligence with survey data.   The new system:

• Puts powerful data mining, survey data analysis and reporting in the hands of everyday business managers
• Dramatically reduces the time and cost of creating usable data - from weeks to hours
• Allows easy configuration and sharing of executive-level dashboards with drill-down capability
• Enables easy online report sharing and hierarchical report management

Feedback Intelligence is available as a new module in our enterprise feedback management suite.  Watch our two-minute tour or sign up for a demo!

Humility and Data Analysis

Over the years, I've read lots of discussions about how to analyze data, with many explanations of statistical methods, yet never once have I read anything that encouraged  analysts to set aside their worldviews.  As we in the United States anticipate tonight's Vice Presidential debate, I can't help but recall last week's Presidential debate, where CNN showed the reaction of Republicans, Democrats and independents as each candidate spoke.  I'm sure to no one's surprise, when Obama spoke, Democrats approved and Republicans disapproved;  when McCain spoke, Republicans approved and Democracts disapproved.

Now comes new science that says that the political party a person belongs to affects how they analyze information, especially how they analyze incorrect information (When Corrections Fail: The persistence of political misperceptions and The Enduring Importance of False Political Beliefs).  Partisans are less likely to believe factual criticisms of their candidate, and more likely to believe unwarranted criticisms of their opponents' candidate. Having the research to back it up is great, as it's something I've always intuitively believed. 

This is one reason why I am a political independent.  Being neither a Republican nor a Democrat has made watching the dialogue at Election.Twitter.com especially enlightening for me.  I've seen lies and half-truths championed by partisans on both sides, yet spoken by people who clearly believe what they are saying. 

How does this relate to survey research?  People-with all their biases-write surveys, and people-with all their biases-analyze surveys.  This presents many challenges for market research.  The world view of the survey author, the survey analyst and the decision maker all need to be kept in mind.

• Survey Author - In questionnaire design, obvious leading questions are easy to spot: "Mayor, which is your preferred method of stifling dissent, banning books or burning them?"   Unintentionally leading questions are harder to spot; sometimes the survey author writes a question using terms he or she considers neutral that aren't neutral.  For instance, "Are you pro-choice?", "Are you pro-life?" or "Do you support abortion rights?" are all leading questions.  Instead, look at how the Harris Poll approaches this sensitive subject:  "The two main groups in the abortion debate are the so-called pro-life group, which opposes abortion, and the so-called pro-choice group, which supports women's right to have an abortion. Which one of these groups do you tend to support more?"
• Survey Analyst - Sometimes, the results to a particular question are so surprising that the analyst can't believe them.  Occasionally, this is for good reason, because the data is wrong:  the question was ambiguous or the choices miscoded or the scales reversed or some other mistake has been made.  Frequently, though, the data is right, but surprising.  Too often, in this case, the analyst will look for reasons to explain away the data rather than embrace the insights provided by this surprise.  From early work I did with Bill Ablondi on mobile professionals, I remember being surprised that professionals with the most interest in the product concept of a PDA were not, as Apple was relentlessly championing, people who did not yet use computers.  Even though Apple was crafting its entire Newton strategy around that market, the greatest interest in fact was from people who already had computers and now wanted access to digital information from their computers in a more portable form.  In hindsight, this is obvious, but it was not obvious at the time, when I was caught up in Apple's reality distortion field (a great phrase for the corrosive effect of world views).
• Decision Maker - Nothing is more disheartening to the analyst then presenting data to the client or decision maker and having it rejected outright.  The client literally refuses to believe the data.  I recall one product-planning project where we took the respondents' reported interest in the solution at any price, multiplied that by purchase likelihood at different prices, and built it into a financial model of likely sales over time.  We were not saying launch or don't launch the product, just reporting our estimate of sales.  The client was furious that this analysis significantly underrepresented the opportunity.  We had verbatim comments to back up the analysis, and stood by it.  The client ignored the advice and built and launched the product anyway.  Its sales didn't even reach our forecast. 

All the best practices of survey research and all the proven methods of statistical analysis are useless if the survey author, analyst or decision maker lets their world view get in the way.

If many Republicans see the world through rose-colored glasses, and many Democrats see the world through blue-colored glasses, tonight, when we watch the debate, we should each try to see it from both angles.  That might help us replace a potentially two-dimensional view of the event with a richer, three-dimensional view.  Because often times the analyst is as important as the analysis.

Customer Service Analytics: Adding Intelligence to the Service Experience

At the Gartner CRM Summit 2008, Gartner analyst Gareth Herschel discussed Customer Service Analytics: Adding Intelligence to the Service Experience. He outlined four best practices:

1. Define scope of analysis: Is your organization analyzing the service process in its entirety or the IVR/call-routing system or the call center staff?
2. "Good, inexpensive call" may be an oxymoron: A call that is too long costs more and annoys the customer; a call that is too short may be missing an opportunity for engagement with the customer and may not resolve the issue at hand.
3. One size doesn't fit all: Don't judge all calls by the same criteria. Categorize calls by the purpose of the call and the value of the customer, and then develop metrics for this segmentation.
4. Tailor results to different views: Too often analytical reports don't really meet the needs of anyone in the organization. Some reports need to focus on "exceptions not averages", and should include alerts and triggers so that the company can take action to intervene on behalf of dissatisfied customers; some reports need to focus on the big picture, providing "context not summaries" and should try for advanced visualization. Too often dashboards have become just a collection of metrics; the best practice for dashboards is to compose them of "related metrics that paint a whole picture when presented adjacent to one another."

Gareth made the point that companies should pay special attention to "bad news". He quoted Despair.com's service department: "We're not satisfied until you're not satisfied." He said, while it was tongue-in-cheek for Despair.com (a publisher of "demotivational" posters), it was painfully true of many firms, who - too often - make providing negative feedback an arduous process, "putting the complaining customer into a penalty box". Negative feedback can often lead to the best insights.

For capturing market insights from service interactions, Gareth displayed the following quadrant analysis:

	Mature Analysis Strategies & Techniques	Immature Analytical Expectations & Best Practices
Data We Need to Collect	EFM	Blogs
Data We Already Collect	Enterprise Data Warehouses	Customer Call Recordings

Blog and forum feedback can be "unreliable, prone to manipulating, and unquantified". Gareth suggested using it to identify potential service issues that were then quantified using enterprise feedback management.

Analysis of customer-service issues can be applied in three different ways:

                                                                               

Concept	Tactic	Role of Analysis	Analytical Techniques
Pre-emptive: Customer Issue Avoided	Identify & Resolve Causes of Problems	Identification of High Cost Issues	Cost Allocation       Call Categorization       Root-Cause Analysis
Pro-active: Customer Notified Issue is Being Addressed	Resolve Issue on Corporate Schedule	Identify Treatment Strategy	Issue Detection
Re-active: Customer Calls to Resolve Issue	Divert Issue to Self-Service Channels	Divert Issue to Self-Service Channels	Intelligent Call Routing

Gareth concluded with a complex slide showing categories of service analytics vendors. He pointed out that no suite solutions exist yet. Over time, different technologies will become bundled together and consolidation will happen. He said that enterprise feedback management and text analysis were a natural fit, and he expects more bundling of such solutions in the future (Clarabridge and Vovici have been doing just that since July). He doubted that 10 years from now the dozens of technologies he showed would be bundled into a standard customer-service analytics suite. He said, if it did happen, there would be another 20 new categories of analysis that would have emerged and would not be bundled in the suite!