Survey Translation from 30,000 Feet

Nancy Porte, VP of customer experience for Vovici, presented this morning to the Technology Services Europe conference in Barcelona on "Joining Voices:  Understanding and Leveraging Multi-National Customer Feedback". One of her key points was that, too often, novice researchers think of translation as a front-end task in the survey design process, when in fact the needs of translation must be considered at each stage of the survey process: Study Design, Survey Design, Fielding and Analysis.

• Study Design - After you decide which languages you need to conduct research in, hire the appropriate translators and develop a conservative schedule that takes into account the many pitfalls that await the international researcher.
• Too often, translation is "insourced" to employees who do not have the requisite experience. The best translators have an excellent understanding of the originating language, the target culture, the industry and survey research itself, since survey translations are more demanding than traditional translations. Find and contract with native speaking professional translators (preferably certified translators) where possible.
• The biggest mistake in designing studies with translation is creating unrealistic schedules. Budget for lots of extra time: time to translate the final questionnaire into each target language, extended time in the field to account for local holidays, and time for translating and post-coding open-ended comments.
• Survey Design - A common mistake is to finalize the questionnaire in the source language and then send it off to be translated into the target languages. Instead, you need to design a master questionnaire that will be localized, not just translated. Structure it for translation:
• Avoid jargon, slang and technical terms. Rewrite for readability.
• Use as few open-ended questions as possible.
• Use closed-ended choice lists tailored for local markets and brands.
• Use country-specific skip patterns since entire sections of the questionnaire may differ, depending on the structure of the industry being studied in that country.
• Make certain to prepare back-translations from the questionnaires to ensure quality and accuracy. You can also turn to panels of respondents to assist with translation, as InSites Consulting is pioneering.
• Fielding - When inviting people to complete surveys by email, send them an email in their language. If your database list does not specify their language, or you are concerned about the accuracy of your data for that field, send them a generic language picker with a unique link for each translation. While you can set online surveys to automatically route to the appropriate translation based on the primary language set in the browser, respondents' browser settings do not always match their native language.
• Analysis - While fielding a multilingual survey, one of the best feelings in the world is analyzing in real-time the results to closed-end questions. Thanks to your earlier work translating the questionnaire, you can immediately chart the answers as they pour in despite being completed in dozens of different languages (assuming you have the right survey software). Open-ended questions, of course, take more time to analyze. During fielding, you can use machine translation to get the gist of what respondents are saying, but for accurate final analysis rely on human translators. By all means compare and contrast behavioral results across markets and languages, but take care with attitudinal data, which can often be subjectively different-better to use that information for exploring attitudes of different segments within the country or for establishing a baseline for future trending.

Nancy gave the case study of the multilingual surveys of one airline, a Vovici customer with over 3000 flights a year to over 200 international destinations. The airline fields a satisfaction survey in ten different translations: Chinese (two dialects), Dutch, English, French, German, Korean, Japanese, Portuguese and Spanish. Local translation companies in each market were used for the questionnaire, which measures satisfaction with flights and with the frequent flyer program. Over 100 email triggers are used depending on what passengers are dissatisfied with. The survey was fielded in four weeks and initially ran well in every country except Japan, which suffered from suboptimal response rates and high abandonment rates. In the past year, over 2.2 million responses have been collected and analyzed, informing decisions about seat selection options and benefits of the frequent-flyer program. As a result, the program is now being rolled out for general passenger satisfaction research.

When conducting your first multilingual study, make certain to consider the impact of translation on the entire process, not just questionnaire design. Of all the research projects you will do, multilingual studies will require the greatest coordination of resources and general project management skills. With proper planning, though, your multilingual survey will have a smooth flight!

Survey Emails FAQ

Here are links to some answers to frequently asked questions about emailing invitations to online surveys.

• What should I keep in mind when renting email lists for surveys? When renting access panels?
• How do I create a representative email list of my customers?
• How do I make sure our invitation isn't flagged as spam?
• How do I entice recipients to open the survey invitation?
• How do I write compelling survey invitations that will prompt people to take the survey?
• What are best practices for unsubscribe requests, to let recipients of survey invitations opt out of mailings?
• Do I need to offer survey incentives? If so, what kind?
• What's the most important way to improve response rates?
• What kind of response rate can I expect?
• How concerned should I be about low response rates?

If you have an unfrequently asked question about survey email campaigns, please leave a comment, and I will get you an answer.

SharePoint as Survey Software

We often have prospects looking to migrate to a survey software application from a home-grown system. Recently, we've been seeing more organizations looking to move from Microsoft SharePoint, an enterprise content management platform.

What clients liked about SharePoint when it comes to building online surveys:

• The WYSIWYG form editor makes it easy to build basic surveys.
• It offers complete flexibility for programmers to use HTML, JavaScript, AJAX techniques, Flash and other standard web development approaches.
• It stores data directly in the repository that the administrator selects (e.g., SQL Server or Oracle).
• Survey results are readily exported as CSV (Comma-Separated Value) or XLS files.

Here are the features that caused SharePoint users to move to a dedicated feedback platform:

• It is difficult for business users to create surveys of moderate or significant complexity with SharePoint.
• The lack of integrated email support meant that invitations, reminders and thank-you emails had to be created outside the system.
• The lack of panel management meant administrators couldn't target respondents and track past survey activity.
• It is difficult to personalize surveys based on known data.
• SharePoint lacks survey-specific web services, making integration more tedious.
• Some users reported security concerns.
• Until recently, there was no support for basic skip patterns (available with MOSS 2007) or advanced branching.
• It has no real-time reports or dashboards and can't export results to SAV or PPT files.

SharePoint is a great interim solution for organizations first adopting web surveys. It excels at short, uncomplicated and static surveys, but organizations outgrow it when they need advanced questionnaires, intermediate MR functionality, integrated email invitations, panel management or enterprise feedback management.

 Experience Vovici    Request a Call from a Vovici Rep     Request a Demo with a Vovici Rep

Best Practices in Mobile Research

Chris Ferneyhough and Sonia Bishop of Vision Critical discussed best practices for fielding online surveys to mobile audiences in the ESOMAR Online Research 2009 conference. Chris began by pointing out that mobile Internet adoption outpaces desktop Internet adoption and forecasts that eventually usage of the mobile web will be 10 times usage of the traditional web.

Vision Critical asked respondents to an online survey if they had received their email invitation on their phone: 1.9% had in the US, 1.2% in UK and 3.8% in Canada. Clearly, respondents are already completing online surveys on mobile devices, even though authors have often not taken this into account. Chris mentioned that a panel registration survey for a smart phone vendor didn't actually work for that smart phone, because of the registration form's reliance on JavaScript, which was off by default on the phone: the open-ended "Other (please specify)" box was locked and disabled because it required JavaScript to enable it once the corresponding radio button was clicked. Since client-side scripting is disabled on many phones, the data your survey collected may be wrong.

Researchers need to recognize the fact that online surveys are being completed on mobile devices and need to be optimized for that medium. The wide variety of smartphones at GSMArena.com reveals hundreds of different models with dramatically different market share in different countries. Colors and fonts are implemented differently by different phones and may not be implemented at all on a smart phone: a certain color may render some text unreadable.

Many respondents are unfamiliar with their web browser or alphanumeric entry mode on their phone's keypad. Many respondents are concerned about data costs: some have unlimited data plans, others have pay-as-you-go plans. Many have low data connection speeds.

To do further research on these topics, Vision Critical studied respondents who are smartphone users and are willing to complete questionnaires on their phone. The sample was balanced and weighted on gender and age, with 500 Canadian respondents, 118 US respondents and 107 UK respondents. The survey covered attitudes towards the national economy.  No significant statistical differences were found for closed-ended questions on mobile devices vs. desktop devices. For open-ended questions, of course, desktop users were more verbose.

The likelihood to participate in future surveys on mobile phones was greatest for iPhone users (47% were likely to), compared to only 34% of Blackberry users and 23% of all other smartphone users.

Sonia presented back-end mobile research best practices:

• Maximize use of the available space
• Profile your panel for smartphones, whose email addresses may vary for the phone vs. the desktop
• Identify devices and models supported by your data collection software
• Manage the process of deploying surveys to mobile panelists

And questionnaire design best practices:

• Use simpler question types
• Avoid Flash effects and JavaScript validation
• Write more concise questions and answer lists to minimize scrolling
• Put the Next button "above the fold"
• Limit survey length to 10-15 questions (unless heavily incentivized)
• Keep it simple: avoid color, grids, images, etc.
• Develop for the lowest-common denominator devices

Clearly, survey authors fielding online surveys need to take mobile users into account when developing surveys.

Survey Testing Case Study

Survey Test Mode

In a similar way, many users of our survey software dislike the fact that the first time they publish a new survey, it goes into test mode.  This is a not-so-subtle reminder that they should double-check the survey before inviting participants to respond.

Self Test

Pre-Test

That last question is particularly hard to answer in a self-test.  When I am feeling very unsure of a study, or the results are strategic rather than tactical, I will pre-test it on coworkers or, even better, on a small sample of the target audience (no more than 50).  I will end the survey with some questions about the survey itself, to identify areas or survey structures that were confusing or ambiguous. 

Pilot Test

Publish

Common Rating Scales to Use when Writing Questions

One of the most frequent mistakes I see when reviewing questionnaires are poorly written scales. Novice survey authors often create their own scale rather than using the appropriate common scale. It's hard to write a good scale; instead you are better off rewording your question slightly so that you can use one of the following.

Acceptability	Not at all acceptable, Slightly acceptable, Moderately acceptable, Very acceptable, Completely acceptable
Agreement	Completely disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Completely agree
Amount of Use	Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness	Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate
Awareness	Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware
Beliefs	Not at all true of what I believe, Slightly true of what I believe, Moderately true of what I believe, Very true of what I believe, Completely true of what I believe
Concern	Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned
Familiarity	Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar
Frequency	Never, Rarely, Sometimes, Often, Always
Importance	Not at all important, Slightly important, Moderately important, Very important, Extremely important
Influence	Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential
Likelihood	Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely
Priority	Not a priority, Low priority, Medium priority, High priority, Essential
Probability	Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable
Quality	Very poor, Poor, Fair, Good, Excellent
Reflect Me	Not at all true of me, Slightly true of me, Moderately true of me, Very true of me, Completely true of me
Satisfaction (bipolar)	Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied
Satisfaction (unipolar)	Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied

This list follows Krosnick's advice to use 5-point unipolar scales and 7-point bipolar scales.

Let me know any of your favorite scales that I omitted.

Hurdles in Race to Turn Recipients into Respondents

You've chosen the list of people you want to invite to your survey.  You still have a lot of hurdles to jump to get each recipient to actually take your survey:

• Make sure their email address is not on the unsubscribe list
• Keep your survey invitation from being flagged as spam
• Craft a survey subject line and From field that help get the invitation opened  
• Cover the six points important to respondents in an invitation
• Write an invitation that is compelling enough that recipient clicks on the link to take the survey

Representative Web Surveys Require Good Email Lists of Customers

Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.

Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don't, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.

Organizations with unrepresentative email lists need to take the following steps for web-survey success:

• Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
• Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
• Conduct a telephone study or paper survey of customers for which your organization doesn't have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
• Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication. 
• Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.

What steps have you taken to make sure that your web surveys are representative of your customer base?

Getting Your Survey Invitation Opened

Your survey invitation gets past the spam filters and shows up in someone's email inbox. Now the recipient will have two or three items at hand to decide whether to click Delete or to open your message:

• From field - The From field consists of the sending email address and the alias for that address. Nothing will get your email deleted faster than having an impersonal email address without an alias, such as - a particular favorite! - noreply@example.com. Always include the alias. What type of alias works best varies by organization; examples to experiment with for your invitations include:
  • The name of an individual - The name of an employee managing the survey or commissioning the survey can be very effective.
  • The name of the brand or company - While this can be impersonal, it may outperform the names of individuals for B2B vendors.
  • Brand name + "Surveys" - For some organizations a From field such as "Acme Surveys" works best.
  • Hybrid - Some panel companies test combinations such as "Acme Surveys & Jane Doe".
• Subject line - The text of the subject line will have the biggest impact: "Your Feedback Needed", "Some Quick Questions for You", "Help Us Serve You Better", "Learn What Your Peers Think About -----" are all subject lines that have been successful for survey invitations. Your organization will need to experiment to determine what works best for your audience.
• First line of the invitation - Some email programs display the first line of the invitation in grayed-out text immediately after a bold subject line. All the more reason to pay special attention to the copy you use in the body of your invitation.

What tactics have improved your organization's open rate on surveys?