Surveying the World on 16 Dollars a Day
Posted by Jeffrey Henning on Sun, Apr 11, 2010
International surveys may be the type of survey-research projects most likely to go over budget, costing you more dollars, pounds and Euros then you planned for. As I think back over 22 years of research studies, the majority of project overruns involved fielding a survey in multiple countries. If your client (internal or external) makes any of the following statements, prepare to be in the red!
"I thought we could use free software to do the translation." You could, but many of your questions would be ungrammatical gibberish. While Star Trek may have had a Universal Translator, we're going to be able to beam ourselves through walls before a device translates from any language to any language with high accuracy. You need humans to create accurate translations of surveys, which brings us to the next sentence you'll dread hearing.
"We're going to have our local offices do the translation." Yes, coworkers in international offices qualify as human, and they certainly understand their local market. What they do not necessarily understand is the art of translation or the specialized language of surveys. They may also have great intentions but have difficulty getting the translation done in a timely fashion. The ideal survey translator is a professional with extensive experience translating questionnaires, a background in the industry being studied and knowledge of the local market. Since this ideal is often outside the budget, the next-best solution is to pair the translator with your in-country coworkers.
"We've finished the U.S. version of the questionnaire. Now we just need to get it translated." A common mistake is to think of this as fundamentally an exercise in translation, instead of an exercise in localization. Your country manifests itself differently in every market - you have different products, sometimes just subsets of your global product line but there are often country-specific products. Your competitors vary by country. Pricing varies by country. Develop a master questionnaire, in English, that contains localization instructions. Provide notes to the translators to further specify your intent.
"We don't have the time or money to re-translate the translated questionnaires back into English." The time to catch a mistake is before you field the survey, not after you get back results that you don't understand. David R. Morse, president of New American Dimensions, gave one example at the MRA First Outlook Conference. David once was surprised to see in a survey that 100% of Spanish-speaking respondents disagreed that "the Internet is color blind". Rechecking the Spanish translation, he found that it had been translated literally: "the Internet has red-blue-green color blindness". English questionnaires are often informal and use idioms and expressions that do not easily translate; English, like most languages, has words with dozens of different meanings, making a question that wasn't ambiguous to you ambiguous to your translator, who misinterprets it. You need to budget for back-translation of your questionnaire or risk refielding portions of your research.
"I can't imagine it taking more than six weeks." Let's say your international survey project involves six different markets. That provides six times the opportunities for delays. I've seen delays because of the variability in the availability of skilled translators for a particular language, as well as delays in fielding due to country-specific holiday schedules (especially over the summer in different European countries). It is easy to run over budget when you dramatically underestimated the time required.
"I didn't really expect to have all these verbatim responses." Design the questionnaire up front to minimize open-ended questions. Nothing is more magical than getting back results to a closed-ended question from 10 different languages and being able to instantly analyze the results! Nothing is more frustrating than realizing that cost overruns have left you with no money to translate all of the verbatim responses.
"Here is a chart comparing satisfaction in each major market." That's like comparing apples to naranjas. You are going to be asked to compare the ratings in different countries, but I contend that the analysis is meaningless. It can reflect differences in how questionnaires in general are answered in different countries more than underlying actual differences in satisfaction. If you are fielding a significantly larger sample size over a long enough time period, compare early responses to late responses to provide trend analysis within each market.
"Wow! I can't believe we got that done under budget." I confess that I have never heard those words used to describe any international survey research. Prepare your budget conservatively and factor in what the French call faux frais, "items one overlooks when making a budget". Then you might have some extra dollars, pounds or Euros on hand when the project ends!