Welcome to the Listening Post!

Mode Effects: Same Survey, Different Medium, Different Results

Tweet

 

Respondents do not answer questions the same way in person, on the phone, on paper or via the web. Different survey modes produce different results. Robert Groves, in his 1989 book Survey Errors and Survey Costs, argues that each survey mode puts respondent into a different frame of mind (a mental “script”).

• Face-to-face surveys prompt a “guest” script. Respondents are more likely to treat face-to-face interviewers graciously and hospitably, leading them to be more agreeable. And, of course, differences between interviewers administering the survey can lead to a range of interviewer effects on survey results.
• Phone interviews prompt a “solicitor” script. Respondents are more likely to treat phone interviews the way they treat calls from telemarketers, making them more likely to “satisfice” (go through the motions of answering questions) in order to get the interviewer off the phone. A wariness of who may be on the other end of the phone also leads them to provide more socially acceptable answers than in any other survey mode.

Let’s extend Grove’s ideas to methodologies he didn’t have to worry about:

• IVR interviews prompt a “voice mail” script. Respondents may hurry through an IVR survey, in the same manner they attempt to rapidly navigate a voice-mail system. Some research has even shown that the shape of the standard telephone keypad leads to different numbers being chosen more often in rating scales than in other modes.
• Internet surveys prompt a “web form” script. Respondents may easily slip into the mindset of filling out web forms, which are often used for online transactions such as creating user accounts or purchasing goods and services. Such forms are often skimmed rather than read, and eye-tracking studies of surveys indicates that question text is often skipped as respondents’ eyes jump to the choice list.
• Mail surveys prompt a “form” script. Respondents may complete paper surveys received through the mail much like they fill out forms at doctor’s offices and banks, with careful attention to detail.

Studies of mode effects are sometimes contradictory but a general pattern emerges. Social desirability bias tends to be highest for telephone surveys and lowest for web surveys:

1. Telephone surveys
2. Face-to-face surveys
3. IVR surveys
4. Mail surveys
5. Web surveys

This makes web surveys particularly well suited for studies of sexual behavior, illicit activities, bigotry, personal health and other sensitive or threatening topics.

Mode effects can be complex and subject to interactions between differences in respondent demographics, subject matter and mode. 

Take care when comparing the results of surveys conducted by different modes. Humans process language differently when reading, when listening to someone over the phone, or when listening to someone in the same room (when visual cues and body language kick in). It’s no surprise that these different modes lead to different behavior by respondents.

Reg Baker of Market Strategies International occasionally writes about the latest research into mode effects on his blog, “The Survey Geek”. Wikipedia lacked an article on “mode effect”; for those of you familiar with the topic, please contribute to the new article I created.

See also:

• Social Desirability Bias: Sex, Drugs & Rocking Surveys
• Using Interviewers to Administer Web Surveys