Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
|
 RSS Feed
Posted by Jeffrey Henning on Wed, Nov 11, 2009
 Jeremy Whyte, director of customer feedback and reporting with Oracle Corporation, presented details of Oracle's extensive Voice of the Customer research program to the American Marketing Association in a research webinar on October 20. Surveys form the foundation of Oracle's Voice of the Customer program: listening to customer input through Vovici surveys provides "comprehensive feedback across the Oracle ecosystem and customer ownership lifecycle". Oracle conducts hundreds of surveys, grouped into relationship surveys, transactional surveys and targeted surveys.
- Relationship surveys with customers, partners and employees are the most strategic surveys, highlighting customer experience and loyalty drivers across cumulative contacts. Why survey employees in a Voice of the Customer program? Because Oracle has identified a correlation between employee satisfaction and customer satisfaction and prioritizes investments to improve employee satisfaction based on how that will drive increased customer loyalty.
- Transactional surveys measure the quality of each service response by organization. Short term these surveys trigger immediate action on a customer-by-customer basis, and long term these drive operational improvements to improve service quality. Transactional surveys are conducted for technical support, customer service, consulting services, education services, sales win/losses, implementations and even M&A impact on customers.
- Targeted surveys are used primarily for qualitative research to supplement relationship and transactional surveys. Sample applications include competitive intelligence, product and service planning, marketing, referencing, user group satisfaction and general market research.
All of these survey results are combined with operational measures and financial outcomes in a customer satisfaction linkage analysis. As a foundational platform, taken together these surveys empower Oracle's wider Voice of the Customer program, which reaches beyond surveys for other additional types of feedback.
How effective has this been? Oracle has improved customer satisfaction each fiscal year for five years running and its customers now demonstrate the greatest propensity to recommend that they ever have. If you talk to a large Oracle client, you will hear firsthand how they have seen Oracle adapt and improve to serve them better. To achieve similar results for your own organization, make surveys the foundation of your efforts.
Posted by Brian Koma on Fri, Oct 30, 2009
 Are you spooked by poor survey response rates? Do you get a cold chill when your boss asks what can be done to make your surveys better? Are there skeletons in your survey closet you'd rather avoid? Are you bedeviled by survey bias, poor response rates and bad reporting? By understanding the problems that haunt most surveys, you can enhance the quality of your efforts, dramatically improve the value of your data and ensure high participation rates.
See if you recognize any of these goblins, ghosts and gremlins:
- Zombie Surveys - In the movie "Shaun of the Dead," zombies move forward relentlessly under their own power, but have no thinking behind them. Zombie surveys are typified by survey projects that occur year after year because "we've always done it that way." Since good surveys start with good goals, getting rid of zombie surveys means asking two critical questions: Do I really need to conduct this survey to get this data? What action am I going to take with the data I gather? If you can't answer these questions, you've got a zombie to kill.
- Frankenstein Surveys - Dr. Frankenstein bolted a monster together out of unrelated parts, and many organizations create surveys the same way. Too often individual departments are asked to contribute questions to a survey, resulting in an out-of-control monster. Avoiding Frankenstein surveys requires you to relentlessly narrow the scope of your survey and focus only on the data you need to make business decisions. Don't stitch together too many survey questions, but use a scalpel to cut out as many questions as possible.
- Jekyll & Hyde Surveys - Dr. Jekyll looked perfectly normal, but within minutes of drinking his potion he turned evil. Jekyll & Hyde surveys look normal at the beginning, but quickly turn bad by injecting biased questions or by skewing response scales to summon a pre-ordained result. Never ask questions in such a way that respondents can determine where you stand on any topic. You can avoid drinking Dr. Hyde's potion by striving to write objective questions.
- Response Rate Ouija Board - Conjuring up a high response rate requires more than just a dark room and the right incantations. The right ingredients for great response rates are:
- Vampire Invitations - Bats can reach their destination even in complete darkness, but some of them turn into bloodthirsty vampires. Getting invitations out to potential respondents can involve a similar transformation that can limit your ability to field the survey if you're using non-permission based lists, or more importantly, not complying with the CAN-SPAM Act of 2003. Repelling Vampire Invitations means your e-mail must be viewed as a friendly spirit by following these simple guidelines:
- E-mail text contains the physical street address of the sender
- Subject line is accurate and does not mislead the invitee
- "From" line contains the name of the company or representative
- Content includes a valid opt-out or unsubscribe link
- Email list has been reduced by removing names on your suppression list
- Headless Horseman Reporting - If you're developing mindless reports that get buried and ignored, you're a victim of Headless Horseman Reporting. Since insights are the reason you conducted a survey, you've got to concentrate on survey reports that people will be dying to read. Overcoming this particularly pernicious gremlin means that you mustn't feel compelled to just report on data in the order it was gathered. Call out the most important elements no matter where they were collected in the survey. Make sure your report addresses the Essential Question that inspired the research in the first place.
- Silence of the Lambs - Hannibal Lecter may have done despicable things, but he wasn't shy about talking about them ("I ate his liver with some fava beans and a nice Chianti"). You too must talk up your work. Consider developing summary reports, web seminars or blog posts about your survey data, or ultimately, create an online community to discuss results and show people that you're listening. For the greatest return on your survey investment, engage employees with VOC data, then share your results with customers to close the feedback loop and open up greater participation with your next research.
Exercise these best practices to exorcise the goblins, ghosts and gremlins from your research projects!
Posted by Jeffrey Henning on Wed, Oct 28, 2009
 Steve Lavine of Toluna discussed mobile interviewing at the 2009 ESOMAR Online Research conference, beginning by acknowledging that growth has been slower than expected. While online surveys provide real-time data delivery, such surveys are completed at home or the office: mobile surveys, on the other hand, can now put the survey right at the point of sale.
Mobile interviewing works well when you need immediacy or have to reach younger generations or otherwise hard-to-reach audiences. Mobile interviewing works well for diary studies, where panelists can record impulse purchases as they are being made, eliminating reliance on memory and increasing reporting rates; SMS can be used to send reminders to complete the diary. Camera phones can be used in ethnographic research, as respondents submit audio clips, digital images and even short videos about, for instance, the use of a product. Mobile interviewing can also be used for pharmacological testing.
Handheld devices have long been used for intercept surveys. Mobile tools leverage affordable hardware and work well in developing countries and can provide access to real-time information such as current quota levels.
Good target demographics for mobile interviewing include teens and Generation Y, who are reluctant participants in most other modes of research, but love to text, chat and surf. They love the challenge of the camera phone and eagerly send in images, audio clips and videos; this is true of any income level of teenagers. Much telecommunications research obviously makes sense to do on the mobile phone.
The mobile web provides a rich, extended survey experience with digital images, but many cell phones do not support the web. SMS surveys are discontinuous exchanges of information and are used for simple, text-only polling (a few choose-one or fill-in-the-blank questions), but are accessible to far more people in the U.S. and worldwide than mobile-web surveys. An emerging trend is the downloadable application, which offers rich survey experiences but only to a small installed base; maintaining an access panel of any size or representation is difficult; this works better for a small, well-known group such as employees or partners. IVR surveys fielded to cell phones require the creation and maintenance of an opt-in list for permission contact panelists at the mobile number; invitations are often sent by SMS and the phone IVR survey can be longer, from 5 to 12 minutes in length.
Mobile interviewing provides more rapid response time (2.6 hours) than the response rate of any non-mobile method (5+ hours), often 50% faster. Mobile IVR response time is 3.2 hours vs. 5.1 hours for mobile web surveys. This is a significant benefit for entertainment research. For traditional online surveys, SMS invitations provide a 3.9 hour response time, compared to 6.4 hours for email invitations. Respondents invite by text more closely match the population's age distribution.
In the future, more short-code surveys will be advertised point of sale and upon exiting the store. Expect to see proximity-trigged surveys using GPS, RFID, cell-tower triangulation and other methods, often with panel registration.
Posted by Jeffrey Henning on Wed, Oct 28, 2009
 Bill Blyth, Chair of the ISO TC 225 and Global Methods Director of TNS Global described the ISO 20252 standard at the ESOMAR 2009 Online Research conference.
ISO (International Standards Organization) efforts on survey research grew from European trade associations with concerns about the quality of data collection, dating back to the 1970s for standards for face-to-face interviews in the UK and Netherlands. ISO 20252 covers quality at all stages of the survey process: "As some restaurants in London serving offal say, 'It's nose to tail eating!'"
ISO 20252 addresses the quality triangle of Design, Process, User with Fitness in the center. It specifies procedures and documentation and sets minimum levels of validation for key elements. It applies to subcontractors.
While ISO 20252 covers all methods, including online research, the Access Panel Standard - ISO 26362 - published in 2008 provides an alternative to 20252 for specialist panel providers. The 20252 standard is being rewritten to be technology neutral so that it will not need to be rewritten "every time a new gizmo" comes out.
The ISO 20252 has replaced local standards in Australia, Netherlands, Spain and the UK and will do so in France and Sweden in 2010. Italy and Mexico have 20252-inspired local versions. In total, over 250 companies around the world are certified ISO 20252 compliant; the standard does require a compliance audit by a third party.
ISO is encouraging audited compliance to be a contractual/procurement requirement in RFPs. ISO contends that the ISO 20252 saves adopting organizations money by facilitating getting projects right the first time, requiring less rework.
Since online research is the easiest method of research to conduct globally, global standards are needed to ensure quality. ISO 20252 provides a framework and common language for online quality and is a global standard suitable for any size organization in any country.
Posted by Jeffrey Henning on Mon, Sep 28, 2009
 Nothing is easier for the survey author then to dash off a questionnaire asking respondents to rate a bunch of items on an agree-disagree scale (also known as the Likert scale). For instance:
For each of the following statements, please indicate if you: Completely disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Completely agree.
- My overall job satisfaction is very high.
- The issue of excessive executive compensation is very important to me personally.
- I rarely feel discouraged with my work.
- I am very likely to seek employment elsewhere in the next six months.
You can easily add many other statements to this list for respondents to rate. In fact, I have seen questionnaires with 80 to 100 items, all to be rated on this agreement scale.
Unfortunately, in such batteries of questions, respondents exaggerate their actual agreement. Over 100 studies now have demonstrated acquiescence response bias, as some respondents will agree to almost any assertion. Saris, Krosnick and Shaeffer identify three reasons for this, in their paper "Comparing Questions with Agree/Disagree Response Options to Questions with Construct-Specific Response Options":
- Some respondents are simply agreeable, and indicate agreement out of politeness.
- Other respondents expect that the researchers agree with the listed items and defer to their judgment.
- Most respondents engage in survey satisficing and find that agreeing takes less effort than carefully weighing each optional level of disagreement and agreement.
The standard solution to acquiescence response bias has been to have a balanced battery of items, where each item has a negated counterpart somewhere else in the questionnaire. For instance, averaging the agreement level to "I am generally a satisfied employee" with "I am not generally a satisfied employee" was thought, in theory, to produce a rating that factored out the acquiescence response bias. Saris, Krosnick and Shaeffer put that to the test and found three ways it leads to lower data quality: having to answer twice as many questions, which leads to satisficing; processing negations, which are more complex cognitively; and placing respondents who acquiesce in the middle of the scale.
The solution with the highest data quality does lead to more work for the survey author. Each question needs to be asked with what Saris et al call "construct-specific response options": in other words, a rating scale that can be used to measure the item in question. Applying this recommendation to the four questions above yields:
- How would you rate your job satisfaction overall? Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied?
- How important is the issue of excessive executive compensation to you personally? Not at all important, Slightly important, Moderately important, Very important, Extremely important?
- How often do you feel discouraged with your work? Never, Rarely, Sometimes, Often, Always?
- How likely are you to seek employment elsewhere in the next six months? Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely?
 At first glance, this looks like more work for the respondent, who must read the choice list for each question. While there is more to read, there is less to think about. For the statement, "I am generally a satisfied employee", respondents might have come up with four reasons to disagree:
- They are generally dissatisfied.
- They are neither dissatisfied nor satisfied.
- They are often satisfied, but not often enough to classify it as "generally".
- They are always satisfied, which is more often than "generally".
The agreement/disagreement scale gives the respondent too much to think about for each item and too many potential reasons to disagree. If this particular question were reworded to use a satisfaction scale, respondents are only rating their satisfaction. Less work mentally, and a more accurate answer.
The best practices for agree-disagree scales are therefore simple:
- Avoid them if at all possible, rephrasing each question to use a common rating scale where possible, otherwise using a custom rating scale.
- When the executives or customer sponsoring the research dictate that you must use an agreement scale, use the seven-item bipolar scale: Completely disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Completely agree.
The Likert scale has a long and illustrious history, having been invented in 1932. Sadly, it is now as obsolete as the cathode-ray tube (which RCA first demonstrated could be used as to receive TV transmissions in 1932). Construct-specific response options are the HDTV of the survey world.
Posted by Jeffrey Henning on Tue, Sep 22, 2009
One of the many ways that people use Twitter is to share their good and bad experiences with specific brands. As an example, we analyzed 150 recent tweets about McDonald's and identified four complaints:
While feedback is great, even negative feedback, a tweet-being limited to 140 characters-doesn't provide actionable context. No doubt a McDonald's executive reading these tweets would have some additional questions for customers about their in-store experience:
- Where was the McDonald's that this happened to you?
- When did this happen?
- Did you alert a McDonald's employee at the restaurant to the situation?
- Is there anything else you would like to tell us?
In fact, this is a great opportunity for a transactional survey. A McDonald's social media representative could tweet a reply to each person, along the lines of this invented example: "@VenessaLeal, so sorry to hear about your ice coffee. We'd love additional feedback, to serve you better next time. http://bit.ly/..."
This provides a way to transform unstructured feedback into structured feedback, which would be more manageable. Such a transactional survey would provide McDonald's the additional information they need to take action: they would be able to route the feedback to the appropriate store, where management could remind McDonald's employees of the best practices that would resolve the identified issue. McDonald's could incorporate this feedback flow into their other ongoing research initiatives (e.g., mystery shopping and customer satisfaction studies) to identify ongoing opportunities for improving training and service.
I wouldn't trust automation to identify which tweets are negative feedback, since sentiment analysis is too inaccurate. I think for now a human needs to read the tweets and respond; someone in your marketing and customer-service departments should be reading tweets that mention your brands already. Nor would I extrapolate from the results of such surveys; they are not representative of the wider customer base that you serve, the vast majority of whom are not on Twitter.
As of this writing (September 22, 2009), the @McDonalds Twitter account has issued just 1 tweet: "We'll be joining you very soon. Stay tuned." It will be interesting to see in what ways McDonald's uses its Twitter presence, especially for collecting and managing feedback.
If you are aware of any organizations that use Twitter to initiate transactional surveys, please share in the comments section below. Thanks!
Posted by Jeffrey Henning on Mon, Aug 31, 2009
People initiate satisfaction and loyalty tracking programs for one simple reason – they want to know what their customers think. Maybe about their company, their products, their services or their perceived value being delivered to customers. But, while they may suspect the answers, they are uncertain and therefore they seek the Voice of their Customer. This represents real progress.
Once they decide to kick-off a survey program, they frequently begin to immediately craft the questions. This is where I get very concerned. The process owners acknowledge that they don’t know what their customers are thinking, but they consciously or unconsciously believe they know what’s important to the customer base. Sounds to me like a case of not knowing what you don’t know!
In my opinion, creating a survey program is a five-step process with the first step often skipped.
Step 1 – Identify your target customer’s most important interactions with your business.
Here are two ways to identify the main areas of interest (touch-points): - The cheap and cheerful way – Identify a reasonable number of customers that you believe represent a broad spectrum of your target audience. Talk with them, either face-to-face or by telephone, explain what your company is going to do, and why, and gather their opinion of the most important touch-points that shape their relationship with your company. Make sure to discuss every interaction between the company and the customer.
- The rigorous and more expensive approach – Commission an external organization to conduct a series of focus groups with representatives of your target. The primary advantages of this approach are:
- Unbiased results
- “Better” answers
- All areas will be probed (the knowledge of what to question is a big part of what you are paying for)
- You show customers and employees how serious your business is about this initiative.
On the other hand, of course, it will take longer, and cost more, that doing it internally.
Step 2 – Identify the key drivers of customer loyalty.
Plan a survey program to obtain a statistically valid analysis of the “key drivers” of customer loyalty by doing it yourself or with a company that specializes in satisfaction and loyalty surveys. Based on knowledge resident within your company, preferably at the operational level, identify the elements of each touch-point that your customers responded were important to their relationship with your business. And question the overall level of satisfaction and loyalty and likelihood of additional purchases (the real reason for the whole program).
Survey a statistically valid sample of your target segment or audience. When all results are obtained, perform a regression analysis to identify the key drivers of satisfaction, loyalty and additional purchases. Make certain that the results are valid by looking at the R-squared. From this analysis, the company will have a very good idea about what is important for each key touch-point your customers identified. And it will be able can see which variable, if any, impacts more that one key parameter.
Step 3 – Identify the key drivers of satisfaction for touch-points identified in the previous step.
The objective is to identify the attributes of the selected touch-points that most strongly influence how your customers feel about their experiences with your business’ most influential interactions. Use the same techniques and rigor as in Step 2 since the results will be driving investments and actions. Do not let individual bias guide which attributes you include. Be as sweeping as possible since, once again, you may not know what you don’t know.
Then the fun begins!
Using the results of Step 3, plan a survey program that focuses on those areas with the greatest bang for the buck. As progress is made, begin tackling the lower impact items on the list. Remember to make sure you get enough completed surveys so that the results are statistically representative of your whole target audience.
Step 5 – Re-verify the key drivers identified in Steps 2 and 3.
Unless there are dramatic changes in the companies’ business environment it is unlikely that the key drivers will change drastically in a year. Yes, your company should periodically revalidate its key driver list but you have time to implement improvements first.
And remember, if it isn’t important then why spend the time and money to find out how the customers perceive your performance? So focus your ongoing survey efforts on the items your customers say are most important to their ongoing relationship with your business. Now, at last, you know what you didn't know when you jumped headfirst into the survey pool!
Posted by Jeffrey Henning on Fri, Aug 28, 2009
Most survey researchers pay little mind to the research into the efficacy of different scales, frequently using obsolete rating scales in their surveys: changing scales can be a real bugbear.
That said, here are some well-researched best practices when it comes to scales:
- Use 5-point scales for unipolar scales and 7-point scales for bipolar scales:
"To explore the relation between scale length and reliability, we conducted a meta-analysis of the results of many past studies. Our data consist of results from 706 tests of reliability taken from thirty different between-subject studies. We combined various measures of reliability and various sample sizes, controlling for these and other factors in determining the relation of scale length to reliability. In general, we found that five- or seven-point scales produced the most reliable results. Bipolar scales performed best with seven points, whereas unipolar scales performed best with five." - Jon Krosnick, professor of communication at Stanford, "The Optimal Length of Rating Scales to Maximize Reliability and Validity"
- Use fully labeled scales without showing respondents numeric ratings. Such scales are preferred by respondents and have higher reliability and predictive validity than numeric scales.
- Exclude “Don’t know” and “No opinion” as a choice when presenting your scale.
- The 0-to-10 rating scale for Net Promoter has the lowest reliability and predictive validity of four scales tested.
The above findings are backed up by scientific research. The following best practices, on the other hand, are my personal preferences, for which I was not able to find supporting data:
That said, factoring in the research and my recommendations, this is what I consider to be the best CSAT scale:
What is your overall satisfaction with our company?
- Not at all satisfied
- Slightly satisfied
- Moderately satisfied
- Very satisfied
- Completely satisfied
When a study mixes different lengths of scales, consider standardizing the scales in survey analysis, for instance by mapping scales to a 0 to 10 scale. This can make reports of the results easier to understand. While respondents dislike numeric scales, fully labeled scales are typically analyzed numerically, and the 0-to-10 mapping can aid analysis.
Most organizations fail to standardize on rating scales, making it difficult to compare the results from study to study, from department to department. If you haven’t yet done so, please consider coming up with standard practices to guide your research. To contradict Emerson, when it comes to rating scales, a foolish inconsistency is the hobgoblin of little minds.
Posted by Jeffrey Henning on Wed, Aug 26, 2009
Survey rating scales are either unipolar or bipolar. A unipolar scale prompts a respondent to think of the presence or absence of a quality or attribute: not at all satisfied, slightly satisfied, moderately satisfied, very satisfied or completely satisfied. Statisticians often map these answers to a scale from 0 to 1 (e.g., 0.00, 0.25, 0.50, 0.75, 1.00 for a five-point scale). Where a unipolar scale has that one “pole”, a bipolar scale has two polar opposites. A bipolar scale prompts a respondent to balance two opposite attributes in mind, determining the relative proportion of these opposite attributes. A common bipolar scale: Completely dissatisfied, mostly dissatisfied, somewhat dissatisfied, neither satisfied nor dissatisfied, somewhat satisfied, mostly satisfied, completely satisfied. Statisticians often map these answers to a scale with 0 in the middle: -3, -2, -1, 0, 1, 2, 3. Which type of scale should you use? Wherever possible, a unipolar scale will be the better choice. - It’s less mentally taxing for the respondent: instead of balancing two opposing attributes in mind, they only have to consider one attribute.
- It's more economical, by providing fewer choices: five-point scales are the most valid for unipolar scales, compared to seven-point scales for bipolar scales.
- You’re less likely to get it wrong: a survey author can choose attributes for both poles that are not in fact polar opposites, further confusing respondents. For instance, asking a respondent to rate competitors on a scale from “very affordable” to “very high quality”.
- Many bipolar scales really measure one dimension: if “not at all X” is synonymous with “very dis-X” then use the unipolar scale instead. For instance, “not at all important” is synonymous with “very unimportant”, “not at all appropriate” is synonymous with “very unimportant”, and so on, making those perfect candidates for unipolar scales.
Rarely, though, a bipolar scale may be the better choice. But I’ll save that for tomorrow’s blog post.
Posted by Jeffrey Henning on Thu, Aug 13, 2009
 One of our more popular webinars is The Seven Habits of Highly Successful Surveys. We’ve turned this into a short white paper, and it’s also the basis for my free 73-page ebook, Survey Software Success. Here’s the Cliff Notes version of the seven habits of successful survey researchers (with apologies to Dr. Stephen Covey): - Focus on Your Goal: “Don't let your survey project go astray from the start: be certain to focus on a specific goal. Be precise about what information you need to gather and what you plan on doing with it.”
- Survey the Right Group of People: Use random sampling with these recommendations for sample size if you have thousands or more potential respondents; if you don’t have that many, then attempt a census instead.
- Craft Your Invitation Carefully: You have a lot of hurdles to jump in the race to turn recipients into respondents; make sure your email list is representative, maintain and use the unsubscribe list, get past spam filters, craft compelling subject lines and induce recipients to click that survey link and take the survey.
- Order Questions Logically: Put screener questions first, then open-ended questions, general questions, specific questions, demographics or firmographics and finally any follow-up questions. (Up-front open-ended questions are not appropriate to every survey but are important for needs analysis research, to prevent bias from later closed-end questions.)
- Write Objective Questions: “The saying Garbage In, Gospel Out reflects our willingness to believe computer output, even if it was generated from bad input. Survey researchers are no more immune to this tendency than computer scientists, as poorly worded questions can lead to suspect results and erroneous conclusions.”
- Shorten the Survey: It’s rare that I work on a research project where the questionnaire doesn’t need pruned before being published. Check out this post for six detailed suggestions on how to trim that survey.
- Close the Feedback Loop: “Your respondents complete the survey because they value their relationship with you, and they want to see you improve. Implicit in the fact that you sent them a survey is your intention to learn, adapt and change based on the results.”
OK, so now you don’t have to watch that particular webinar. So go watch one of our other complimentary survey webinars instead!
Error sending email
Email sent successfully
|