Welcome to the Listening Post!

Response Styles Confound Cross-Cultural Comparisons

Tweet

 

Respondents often have bad habits that affect how they answer survey questions. When a survey has a skip pattern that routes to a question that asks respondents why they didn’t give the highest rating, it’s not uncommon to read responses like “I never give anyone a 10” or “I only give the highest score for truly exceptional service”. These respondents exhibit Mild Response Style. (Hurley, 1998, “Timidity as a Response Style to Psychological Questionnaires”.) The opposite style is choosing only the endpoints of a scale: Extreme Response Style. And there are many other response styles, each filtering respondents’ true attitudes.

Response styles are usually ignored by survey researchers, in the expectation that a mix of respondents with different response styles will balance everything out. This is typically true, but it breaks down in cross-cultural comparisons, since response styles differ systematically by culture. For instance, Indians tend towards Extreme Response Style more than Asian respondents, who use the Mild Response Style.

One European study of attitudes towards advertising found no difference in attitudes between Dutch, Danish, French and Portuguese consumers. When the raw data was reanalyzed to factor out the different mixes of response styles in the different countries, a vastly different pattern emerged: Dutch consumers had more negative views of advertising than Portuguese consumers, who had more negative views than Danish or French consumers. Failing to account for response styles led to the survey results being misinterpreted. (Jan-Benedict E. M. Steenkamp & Hans Baumgartner, 1998, “Assessing Measurement Invariance in Cross-National Consumer Research”.)

Different response styles across cultures can have more subtle effects as well. Research at the University of Wollongong in Australia highlights the problem of comparing student evaluations across instructors when those instructors have different cultural mixes of students. Failing to factor out differences in response styles can lead to universities being unable to properly discriminate the quality of instructors when making decisions regarding promotion and tenure. (Bettina Grun, Sara Dolnicar, 2009, “Response Style Contamination of Student Evaluation Data”.)

What are the common types of response styles and how can you counteract them when doing cross-cultural analysis?

Types of Response Styles

Here’s how I contrast the different patterns of response exhibited by the most common styles:

In Optimal Responding, respondents choose the most appropriate answer. For the truncated scales, respondents map their true attitude onto the shortened scale, reflecting in a limited way their actual beliefs. Wrong answers may come through constant agreement or disagreement (Acquiescence or Disacquiescence Response Styles), equivocation (Midpoint Response), impression management (Socially Desirable Responding) or outright lying or carelessness (Noncontingent Responding, not shown).

(I derived the list of common styles from Hans Baumgartner & Jan-Benedict E.M. Steenkamp, 2006, “Response Biases In Marketing Research”; Baumgartner & Steenkamp, 2001, “Response styles in marketing research: A cross-national investigation”; supplemented by Hurley, 1998.)

Counteracting Response Style Bias

Academic literature offers a rich array of statistical techniques to use to factor out the effects of response style bias. If you have an existing survey that is affected by cultural differences and uses rating scales, you should hire a statistician or research analyst familiar with techniques for removing response-style contamination.

If you are creating a new survey, specifically to be used for cross-cultural comparison, the best approaches involve eliminating traditional rating scales and replacing them as appropriate with these:

• Binary scales – As far back as 1946, to minimize response style bias, Lee Joseph Cronbach (of Cronbach’s alpha fame) advocated using two-item scales: yes/no, agree/disagree, dissatisfied/satisfied, describes/does not describe. Essentially this treats everyone as Extreme Response Style responders.
• Choose-many questions – Presenting a list of choices and instructing the respondent to “select all that apply” is an economical form of binary scale, prompting respondents to choose the items they agree with, find important or are dissatisfied with.
• Ranking questions – Another way to avoid traditional response bias is to use ranking scales, where each choice on the scale may be used only once: most important to least important, most satisfactory to least satisfactory, most likely to least likely.
• MaxDiff scaling -- Maximum-difference discrete-choice models are a more sophisticated type of ranking question, typically showing attributes four at a time and asking the respondent to select the best and worst attributes from each set: the attributes with the maximum difference. Research has demonstrated that MaxDiff scaling is superior to rating scales for cross-cultural analysis. (Steve Cohen & Leopoldo Neira, 2003, “Measuring Preference for Product Benefits Across Countries: Overcoming scale usage bias with Maximum Difference Scaling”.)

The worst nightmare for any survey researcher is to reach the wrong conclusions from the data at hand, thereby leading to the wrong decisions. When comparing and contrasting cultures, failing to account for response styles can lead to just those feared wrong decisions.