Welcome to the Listening Post!

Using Interviewers to Administer Web Surveys

Tweet

 

If you've failed to get the optimal response rate to your web survey, despite repeated invitation reminders, you may be tempted to have coworkers call up respondents and administer the survey. They will act as interviewers and read the questions aloud and complete the web form for the respondent.

This is much more complicated than it sounds. Interviewer-administered surveys are very different from self-administered surveys, with different data quality issues from interviewer effects. Further, in this case, potential respondents have already declined to take the survey at their convenience. Good luck calling them and persuading them to take it at your convenience!

Interviewer Effects

A partial list of interviewer effects (behaviors that can lead to quality concerns):

• Rewording a question rather than reciting it
• Emphasizing different words in questions than other interviewers
• Probing inconsistently from one respondent to the next
• Failing to record answers
• Misreading choice lists
• Making other mistakes
• Misrecording responses
• Cheating (e.g., recording a different answer to a choice list in order to leverage a skip pattern to avoid a lengthy block of questions)

Issues with Supplemental Phone Surveys

When supplementing a web survey with phone interviews, here are some issues to concern yourself with:

• Consider outsourcing the interviews to a field house. They have trained professional interviewers and established systems and protocols for maximizing interviewer productivity and quality.
• If you insist on doing the interviews internally, then train interviewers on the questionnaire. Also, make sure to train them together in a single session (for greater consistency across interviewers).
• Since each interviewer introduces individual bias into the data collection, it is better to minimize the number of interviews done per person: share the load; use as many interviewers as possible.
• Assign respondents to interviewers who don't have a relationship with them. A working relationship between the interviewer and respondent will maximize social desirability bias, further skewing the results as the respondent tempers their criticism.
• Keep in mind that the answers of those interviewed by phone will differ materially from web respondents, not just because of the difference in the mode of data collection, but also because this was a group that was more reluctant to be surveyed in the first place.
• There are two schools of thought in interviewing: do it as systematically and consistently as possible or do it as conversationally as possible. Pick one style and stick to it.