Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
Survey Research & Enterprise Feedback Management
|
 RSS Feed
Posted by Jeffrey Henning on Wed, Oct 21, 2009
Jeremy Whyte, director of customer feedback and reporting with Oracle Corporation, presented details of Oracle's extensive Voice of the Customer research program in a Vovici research webinar yesterday, "Driving Business Growth and Profit from a Customer Experience Management Program". Oracle has achieved five years of year-over-year growth in customer satisfaction, even as revenues have grown from under $14 billion in 2005 to over $23 billion as 53 acquisitions have been merged into the Oracle product line. To evaluate and improve customer satisfaction, Oracle has developed a linkage analysis that ties operational measures, transactional satisfaction, customer satisfaction/loyalty and financial outcomes into a coherent model that can be used for ROI analysis:
Operational measures often have a direct impact on transactional satisfaction. For instance, Oracle identified a clear negative correlation between the total time required to resolve a service request and the overall satisfaction with that service request: the longer it took to close the ticket, the less satisfied the customer was.
Transactional satisfaction in turn impacts customer satisfaction. Customers who were more satisfied with service requests were, according to the relationship survey, more satisfied overall with support services and product effectiveness. Such customers also reported higher value received from Oracle and greater loyalty.
Customer satisfaction and loyalty in their turn impact financial outcomes. Oracle was able to identify how much additional revenue for licenses and maintenance renewals would result from increasing the Executive Customer Loyalty Index by one rating point.
Financial outcome modeling lets you link customer behavior to the bottom line, validating the benefits of customer loyalty. Properly refined, the model can be used to predict the ROI of improvements, helping prioritizing those initiatives and guiding resource allocation. For instance, determining how much can be invested in improving response time based on the ultimate financial outcome deriving from greater transactional satisfaction.
For a company as large as Oracle, this is not a one-size-fits-all linkage model. Segmentation is important, as the model varies by customer segment and by product line. Segmented linkage analyses have increased the sponsorship and engagement of senior leaders with the Voice of the Customer research and has proven itself year in, year out, as customer satisfaction with Oracle has steadily increased.
Posted by Jeffrey Henning on Fri, Sep 18, 2009
At this year’s Gartner NA CRM conference, Gareth Herschel presented “Making Gut Decisions More Intelligent”. One of his key points was that most organizations fail to study and improve their decision-making process. “We have to create a culture of auditing the decisions,” he said: not to punish bad decision makers, but to identify ways to make better decisions in the future.
As part of this decision audit, Gareth advocates logging for each decision which best practices were followed. For instance:
- Was the forecast of market demand accurate?
- Was there an opportunity for dissenting voices to be heard?
- Did stakeholders share information?
Then, once enough time has passed to determine whether a decision was a success or failure, plot each best practice on a quadrant.
This quadrant analysis is adapted from "Flaws in Strategic Decision Making: McKinsey Global Survey Results": For the vertical axis, use the percentage of decisions for which the best practice was followed, with 0% at the bottom and 100% at the top; for the horizontal axis, use the difference between the presence of such practices in successful decisions subtracting out their presence in unsuccessful decisions.
The most important quadrant that emerges reveals practices that are “Common in Successes, Rare in Failures”. These are the key decision-making practices that are most important to your industry and organization.
To parallel Kathy Harris’ advice of “Continually innovate how you innovate!”, you should continually analyze how you analyze.
Posted by Jeffrey Henning on Tue, Sep 15, 2009
In last night’s keynote at the Gartner CRM conference, Analytics-to-Action: Key Analyses for Customer-Centric Decisions, research director Gareth Herschel presented a wide-ranging discussion of using analysis to reach conclusions centered on the customer. A number of his points focused on selecting KPIs (Key Performance Indicators).
Gareth related a conversation with a client, after he had asked her about her KPIs. Turns out she had 137 KPIs for her department. “Well, that makes for an interesting definition of K!” he said. As a practical matter, she concentrated on just 7 of the KPIs. Unfortunately, her boss had a different 7 that were important to him. Moral: you have to measure what matters.
Gareth’s advice is that no manager have more than 5 to 9 key metrics. Since we figuratively talk about “keeping balls in the air”, he used the analogy of juggling: world records for juggling a few items measure the feat in hours, while world records for juggling 10 or more items measure the record in catches. Physically and figuratively, we can’t juggle too many items.
Of course, if you are only going to concentrate on a few items, you’d better select them with care. Too often items are tracked because they can be, not because they should be. As Gareth said, “Marketing never saw a piece of data they didn’t want.” Worse, most KPIs are at too simplistic a level and fail to focus on LTV (Life-Time Value).
Customer churn by itself is an example of a bad performance indicator: “80% of value destruction occurs from 20% of your customers; you want to fire those customers,” Gareth said. Telecommunications firms and utilities especially suffer from this. Instead of just looking at minimizing churn rate, for instance, ask “Who’s at risk of churning? Do we care about them? Why will they churn? What should we do to retain them? Would we succeed? Are they likely to be at risk of churning again?” Segment customers by LTV and current value and seek to minimize the churn rate of those with the highest current and lifetime values.
Remember, you can have hundreds of performance indicators but you have to focus on the key indicators to achieve customer-centered decisions.
Posted by Jeffrey Henning on Thu, Aug 27, 2009
One of my frustrations with the Net Promoter Score is that the analyst is supposed to interpret the question differently than the respondent. The respondent is asked a unipolar question, measuring the single dimension of likelihood to recommend an organization: The analyst, however, is told to interpret this as a bipolar scale: likelihood to promote vs. likelihood to detract. A respondent who says they are “Not at all likely” to recommend is treated as a detractor, as is a respondent who says they are moderately likely to recommend (rating of 6). The absurdity of this interpretation is made clear when you ask follow-up questions to the respondents that assume this interpretation. - Why are you likely to recommend against our company? [Asked only of so-called “detractors”]
- “I am not in a position to recommend or not: 5 was neutral so why do I have this question to answer?”
- “That’s not what I said.”
- “I'm never in the position [to recommend].”
- Why are you only somewhat likely to recommend our company? [Asked only of “passives”]
- “A 7 on a scale of 10 is good! Depends on the person's needs.”
- “It is unlikely I would be asked for a recommendation.”
- “Because you made me answer this question before I could finish the survey.”
Since the Net Promoter Scale doesn’t actually ask about detracting behavior, it should not be interpreted as a “net” of promoters minus detractors; at best, it can be interpreted as a Net Very-Likely-to-Recommend Score. How likely is it that you would recommend us or recommend against us to a friend or colleague?
- Extremely likely to recommend against
- Moderately likely to recommend against
- Slightly likely to recommend against
- Neither likely to recommend nor recommend against
- Slightly likely to recommend
- Moderately likely to recommend
- Extremely likely to recommend
So this is one of the rare examples where a bipolar scale is more appropriate than a unipolar scale. That said, the suggestion above is a verbose, confusing question for many respondents. If a client really wants a promoter/detractor segmentation, I do use this question, but otherwise I use a unipolar likely-to-recommend question and steer my client to richer segmentation schemes instead.
Posted by Jeffrey Henning on Thu, Aug 06, 2009
 Reading about the U.S. Post Office's financial shortfall this week reminded me of the following question, which highlights the small role we've played in decreasing the volume of U.S. mail. Roberta, a recent attendee to one of our research webinars, posed the following question: Our customer base strongly skews to an older, retired (and presumably less technically-savvy) demographic. Accordingly, we have continued to use traditional postal surveys to measure satisfaction in this conservative customer base. We are eager to switch to an online survey methodology, but have two related concerns: (1) comparability of historical data (data collected in postal survey vs. online) and (2) switching methodology may effectively 'disenfranchise' some older customers who may not have email access. From a technical standpoint, how serious are these concerns and what, if anything, should we do to address these concerns -- either in terms of survey methdology or at the analytical stage?
If all of your customers are older, then you have less to worry about than if your customers span a wide range of ages. If you have a mix of under-30 customers and over-60 customers, for instance, than switching to the web would change the proportions that responded, which would have a dramatic effect on the results; for such a case, I would advise weighting the results by age segment to reflect the age distribution of customers (see my Age Question blog post for how to ask respondents their age, if you don't already have it).
Our most conservative client migrated data collection methodologies over the course of 12 months, ratcheting up the percent of surveys using the new methodology every month until the end of the first year, at which point all surveys were done using the new method. Each month along the way, they compared the answers of the two groups to see where there were differences and to understand what the causes of the differences were. For analysis, they reported the overall results and the results by each collection mode.
If you did something similar, at the end of the year you could do year-over-year comparisons using only the results collected the year before using the new method.
As to disenfranchising customers, you could position the move to the Web as being done for environmental reasons, to minimize your firm's use of paper and the fuel necessary to ship the mail back and forth. (Hat-tip to Gartner for teaching me how to sell online surveys as 'green'.) You could say that all people would be moved to the new method unless they specifically returned a postcard opting to still receive the surveys by mail. Such a transition to the web is rarely needed for more than a year. Moving from paper surveys to web surveys will: - Lower your costs by eliminating the need for printing and mailing surveys and then doing data entry on the completed surveys
- Dramatically speed up reporting by eliminating the current lag where surveys are in transit to you and then queued up for data entry
- Enable you to set up survey alerts and trigger emails to immediately take action when a customer is unhappy.
Oh, and it is definitely better for the environment! (If not the Post Office!)
Posted by Jeffrey Henning on Wed, Jul 08, 2009
Early in my career I learnt survey data cleaning firsthand from Jo Ann De Clercq, who also taught me how to code responses to open-ended questions. Back then, we had a body of practices that we used from study to study, but no formal documentation of those practices. Data cleaning is emblematic of the historical lower status of data quality issues and has long been viewed as a suspect activity, bordering on data manipulation. Armitage and Berry almost apologized for inserting a short chapter on data editing in their standard textbook on statistics in medical research. Nowadays, whenever discussing data cleaning, it is still felt to be appropriate to start by saying that data cleaning can never be a cure for poor study design or study conduct. Concerns about where to draw the line between data manipulation and responsible data editing are legitimate. Yet all studies, no matter how well designed and implemented, have to deal with errors from various sources and their effects on study results. The authors outline a data-cleaning process with three steps: Screening Phase, systematically looking for problems with the data; Diagnostic Phase, identifying the condition of the suspect data; and Treatment Phase, deleting or editing the data or leaving it as is. 
Screening PhaseExamine data for five different kinds of possible errors: - Lack of data – Do some questions have far fewer answers than surrounding questions?
- Excess of data – Are there duplicate responses?
- Outliers/inconsistencies – Are there values that are so far beyond the typical that they seem potentially erroneous?
- Strange patterns – Are there patterns that imply cheating rather than honest answers? For instance, does a respondent alternate between ratings of 4 and 5 on every other topic in a matrix question?
- Suspect analysis results – Do the answers to some questions seem counterintuitive or extremely unlikely?
Diagnosis PhaseFrom the Screening Phase you have highlighted data that needs investigation. To clarify suspect data, you often must review all of a respondent’s answers to determine if the data makes sense taken in context. Sometimes you must review a cross-section of different respondents’ answers, to identify issues such as a skip pattern that was specified incorrectly. With this research complete, what is the true nature of the data that you’ve highlighted? The five possible values the authors give: - Missing data – Answers omitted by the respondent or questions skipped over
- Errors – Typos or answers that indicate the question was misunderstood
- True extreme – An answer that seems high but can be justified by other answers (e.g., the respondent working 100 hours a week because they work a full-time job and two part-time jobs)
- True normal – A valid answer
- No diagnosis, still suspect – The verdict is out on this “idiopathic” data. When it comes time for the Treatment Phase, you may need to make a judgment call on how to treat this data.
Treatment PhaseYou’ve screened the data and tried to come to a verdict on whether suspect data is guilty or innocent. You have three choices for what to do with suspect data: - Leave it unchanged – The most conservative course of action is to accept this data as a valid response and make no change to it. The larger your sample size, the less that one suspect response will affect the analysis; the smaller your sample size, the more difficult the decision.
- Correct the data – If the respondent’s original intent can be determined, then I am in favor of fixing their answer. For instance, perhaps it is clear from the respondent’s explanation for their ratings that they reversed the scale in their minds; you can invert each of their answers to this question to correct the issue. Some statisticians will argue for imputation, replacing the answers with imputed values, such as the mean for that variable, but the techniques for imputation can become quite elaborate and are best left to professional statisticians.
- Delete the data – The data seems illogical and the value is so far from the norm that it will affect descriptive or inferential statistics. What to do? Delete just this response or delete the entire record? Whenever you begin to toss out data, it raises the possibility that you are “cherry picking” the data to get the answer you want.
However you choose to treat the data, make sure to document in your survey report what steps you took, how many responses were affected and for which questions. ConclusionData cleaning is time-consuming, troublesome and potentially contentious. Further, many issues can be avoided by setting up data validation during the survey design. For instance, recently for an age question, I actually looked up the age of the oldest person alive and used that as my boundary condition. I did double check the results to make sure I didn’t have a surfeit of centenarians answer the survey, but this validation kept someone from entering 1697 as their birth year (as a typo for 1967); had they done so, the survey would have immediately alerted them to the fact. Far better to let the respondent catch and fix their mistakes than have to do it for them. When I started my career, if an answer seemed wrong, we could pull a paper questionnaire to see if a data entry mistake had been made. On the other hand, if it was for a telephone survey, we often had to interpret the response, where the interviewer had clearly misheard the name of a vendor or the acronym used to describe a technology. Web surveys give you the opportunity to prevent many types of errors from being recorded—many, but not all. Cleaning data is never pretty, but it’s an important step that should be taken for any strategic survey.
Posted by Jeffrey Henning on Wed, Jun 17, 2009
 While I believe that the scales with the highest reliability and validity are 5-point unipolar scales and 7-point bipolar scales, and I prefer using fully labeled scales, that is for gathering information from respondents. Presenting information to users of your survey research is a different matter. Simply presenting scales designed for accurate data collection may not facilitate ease of understanding of the results.For instance, since 7-point scales are rare in business use, readers of your report may have difficulty understanding the results if you use a 7-point scale. If you are using a mix of 5- and 7-point scales when reporting results, you are certain to confuse some of your readers. If you are doing a cross-survey analysis where similar questions in different surveys used different scales (see Standardize Your Customer Satisfaction Questions & Rating Scales), you will definitely want to standardize your scales. For presenting data, I typically prefer to map scales to a 0-10 scale. I find that the business professionals I present to intuitively understand this scale. The broader range of values also makes it easier for readers to see differences in the results. For instance, I would typically not present the following, even though this is an accurate representation of the gathered results. Rating of Attributes on a 1-5 Scale
from 1 = Not at all important to 5 = Extremely important | Functionality of product | 4.7 | | Product learning curve | 4.6 | | Quality of technical support | 4.4 | | Ability to grow with product line | 3.9 | | Price of product | 3.6 | | Availability of free trial | 3.5 | | Helpful sales representative | 3.4 | | Third-party reviews of product | 2.9 | | Customer list of vendor | 2.7 | | Vendor’s brand name | 2.5 | On a 1-5 scale, the midpoint is 3.0, not 2.5 (which would be the midpoint of a 0-to-5 scale). To calculate the midpoint, simply average the lowest and highest ratings: for instance, (1+5)/2 = 3. Instead of presenting the above, I standardize the results to a 0- to 10-point scale. Standardization from a 5-point scale is not simply a matter of doubling the answers; doing that would produce ratings from 2 to 10. The 11-point scale has more than twice the granularity of a 5-point scale. Instead, map the scale to a 0-to-1 scale: (X-1)/4 produces a number from 0.0 to 1.0 for each rating. Then map this to a 0-10 scale, so (X-1)/4*10 results in the new scores. Rating of Attributes on a 0-10 Scale
from 0 = Not at all important to 10 = Extremely important | 0-10 Scale | | Functionality of product | 9.2 | | Product learning curve | 9.0 | | Quality of technical support | 8.4 | | Ability to grow with product line | 7.3 | | Price of product | 6.6 | | Availability of free trial | 6.3 | | Helpful sales representative | 6.0 | | Third-party reviews of product | 4.7 | | Customer list of vendor | 4.1 | | Vendor’s brand name | 3.8 | Standardizing bipolar scales typically involves mapping them to a –X through +X range. For instance, from -3 through +3 for a seven-point scale to -10 through +10. Remember, your job as the survey analyst is not to provide an in-depth, detailed view of all the data collected, as it was collected. Giving someone a spreadsheet of all the survey responses would do that. Your job as a survey analyst is to tease out the most important information and present to it your readers in an fashion that will maximize comprehension and understanding. And one tool to accomplish that is standardizing scales for presentation purposes.
Posted by Jeffrey Henning on Mon, Feb 02, 2009
 One retailer takes its core values very seriously, and its most popular survey is a monthly assessment asking each employee to evaluate their local store for compliance with those core values. The climate survey asks questions about trust in employees, fairness at work, care and concern for customers, enthusiasm and day-to-day satisfaction. With about 900 branches, 100 districts and 10 regions, the same survey is viewed over a thousand different ways. Prior to automating this, administrative staff had to painstakingly prepare a thousand reports each month. Talk about doing the same thing over and over again, a la Groundhog Day!
This corporate-climate survey is closely watched by all in management, and their needs are a great illustration of the capabilities of Vovici Feedback Intelligence: - The branch manager and assistant branch manager can drill down on the survey data for their retail location in detail. They can also compare their branch to overall ratings for their district, their region and the nation. They can't view any other branch's ratings, though.
- The district manager can compare each branch in their district and can dive into the details for each branch, looking for what one branch management team is doing better than another. They can compare their district to overall ratings for their region and the nation, but they can't view any other district's ratings and they can't review ratings of branches outside their district.
- The regional vice president and assistant regional vice president can compare districts and branches, looking for best practices that can be applied across the region. They can compare their region to other regions and the nation, but can't view results for other regions, districts or branches.
- Corporate executives get dashboard reports showing high-level trends and comparison of regions and can drill down in detail to the results for any branch, district or region.
Vovici Feedback Intelligence excels at managing the thousands of users and report views that help bring enterprise-wide surveys like this one to life. While implementing Feedback Intelligence won't win you Andie MacDowell, we can guarantee that you will win the love of your staff who are manually compiling these reports day after day after day.
Posted by Jeffrey Henning on Thu, Dec 04, 2008
 Last week we announced the latest product in our enterprise feedback management suite: Vovici Feedback Intelligence. Most EFM and survey software applications take a traditional view of survey analysis, providing frequent reports, verbatim lists and cross-tabulations. While a survey tool can report on each question intelligently, using a pie chart for a choose-one question, a bar chart for a choose-many question, and so on, most survey tools are not meant for ad-hoc analysis. Such tools often make it difficult to manipulate and share survey data. In fact, many of our prospects were exporting survey data to Excel, SPSS or SAS.
The genesis of, and the genius of, Vovici Feedback Intelligence is that it for the first time marries the flexibility of Business Intelligence with survey data. The new system:
- Puts powerful data mining, survey data analysis and reporting in the hands of everyday business managers
- Dramatically reduces the time and cost of creating usable data - from weeks to hours
- Allows easy configuration and sharing of executive-level dashboards with drill-down capability
- Enables easy online report sharing and hierarchical report management
Feedback Intelligence is available as a new module in our enterprise feedback management suite. Watch our two-minute tour or sign up for a demo!
Posted by Jeffrey Henning on Thu, Oct 02, 2008
 Over the years, I've read lots of discussions about how to analyze data, with many explanations of statistical methods, yet never once have I read anything that encouraged analysts to set aside their worldviews. As we in the United States anticipate tonight's Vice Presidential debate, I can't help but recall last week's Presidential debate, where CNN showed the reaction of Republicans, Democrats and independents as each candidate spoke. I'm sure to no one's surprise, when Obama spoke, Democrats approved and Republicans disapproved; when McCain spoke, Republicans approved and Democracts disapproved.
Now comes new science that says that the political party a person belongs to affects how they analyze information, especially how they analyze incorrect information (When Corrections Fail: The persistence of political misperceptions and The Enduring Importance of False Political Beliefs). Partisans are less likely to believe factual criticisms of their candidate, and more likely to believe unwarranted criticisms of their opponents' candidate. Having the research to back it up is great, as it's something I've always intuitively believed.
This is one reason why I am a political independent. Being neither a Republican nor a Democrat has made watching the dialogue at Election.Twitter.com especially enlightening for me. I've seen lies and half-truths championed by partisans on both sides, yet spoken by people who clearly believe what they are saying.
How does this relate to survey research? People-with all their biases-write surveys, and people-with all their biases-analyze surveys. This presents many challenges for market research. The world view of the survey author, the survey analyst and the decision maker all need to be kept in mind.
- Survey Author - In questionnaire design, obvious leading questions are easy to spot: "Mayor, which is your preferred method of stifling dissent, banning books or burning them?" Unintentionally leading questions are harder to spot; sometimes the survey author writes a question using terms he or she considers neutral that aren't neutral. For instance, "Are you pro-choice?", "Are you pro-life?" or "Do you support abortion rights?" are all leading questions. Instead, look at how the Harris Poll approaches this sensitive subject: "The two main groups in the abortion debate are the so-called pro-life group, which opposes abortion, and the so-called pro-choice group, which supports women's right to have an abortion. Which one of these groups do you tend to support more?"
- Survey Analyst - Sometimes, the results to a particular question are so surprising that the analyst can't believe them. Occasionally, this is for good reason, because the data is wrong: the question was ambiguous or the choices miscoded or the scales reversed or some other mistake has been made. Frequently, though, the data is right, but surprising. Too often, in this case, the analyst will look for reasons to explain away the data rather than embrace the insights provided by this surprise. From early work I did with Bill Ablondi on mobile professionals, I remember being surprised that professionals with the most interest in the product concept of a PDA were not, as Apple was relentlessly championing, people who did not yet use computers. Even though Apple was crafting its entire Newton strategy around that market, the greatest interest in fact was from people who already had computers and now wanted access to digital information from their computers in a more portable form. In hindsight, this is obvious, but it was not obvious at the time, when I was caught up in Apple's reality distortion field (a great phrase for the corrosive effect of world views).
- Decision Maker - Nothing is more disheartening to the analyst then presenting data to the client or decision maker and having it rejected outright. The client literally refuses to believe the data. I recall one product-planning project where we took the respondents' reported interest in the solution at any price, multiplied that by purchase likelihood at different prices, and built it into a financial model of likely sales over time. We were not saying launch or don't launch the product, just reporting our estimate of sales. The client was furious that this analysis significantly underrepresented the opportunity. We had verbatim comments to back up the analysis, and stood by it. The client ignored the advice and built and launched the product anyway. Its sales didn't even reach our forecast.
 All the best practices of survey research and all the proven methods of statistical analysis are useless if the survey author, analyst or decision maker lets their world view get in the way.
If many Republicans see the world through rose-colored glasses, and many Democrats see the world through blue-colored glasses, tonight, when we watch the debate, we should each try to see it from both angles. That might help us replace a potentially two-dimensional view of the event with a richer, three-dimensional view. Because often times the analyst is as important as the analysis.
Error sending email
Email sent successfully
|