Total Survey Error: Approaches for Minimizing 9 Common Issues
Posted by Jeffrey Henning on Sun, Mar 28, 2010
Herbert F. Weisberg, in The Total Survey Error Approach, writes:
A new approach to survey research began to predominate by the 1990s: the "total survey error approach." It was always recognized that there are several types of error involved in a survey, but the greatest emphasis was on the type that is easiest to estimate statistically: sampling error... However, sampling error is just the tip of the iceberg in surveys.
Weisberg discusses three groups of errors, each with three categories of error.
Respondent Selection Errors
- Sampling Error - The error from surveying a sample of the population rather than the entire population. Measurable and easily calculated for probability samples, this error can be minimized by increasing sample size. It is unmeasurable for non-probability samples and should not be reported for most online surveys. Don't use non-probability samples when you need to estimate the occurrence of attributes or behaviors in the total population.
- Coverage Error - The error that arises when the sampling approach is not representative of the target population. For instance, landline telephone surveys are only representative of households that have such phones and are not representative of the entire U.S. population. Align your target population and sample; if extensive coverage error exists, narrow your target to reflect the sample.
- Unit Nonresponse Error - The bias introduced when individuals invited to take the survey do not take the survey. What is unknown is how they might differ from respondents. To minimize this error, send repeated reminders to take the survey until a response rate of at least 20% is achieved.
Response Accuracy Issues
Measurement error is the difference between the reported response and the actual true response. Random measurement error is less of a concern than systematic measurement error, which introduces a bias.
- Item Nonresponse Error - The error introduced when respondents don't answer questions at all or select choices such as "don't know". Exclude "Don't know" and "No opinion" as a choice when presenting your scale.
- Measurement Error due to Respondents - Error occurring from dishonest responses, from responses that aren't fully thought through (e.g., from satisficing) and from poorly written survey instruments (e.g., acquiescence bias, questionnaire design effects). Review and shorten questionnaires to reduce this error; avoid incentive strategies that might lead to abuse.
- Measurement Error due to Interviewer - Errors introduced by the interviewer administering the questionnaire (q.v., interviewer effects). In one study, 90% of interviewer errors involved failing to record the respondent's answer. Train your interviewers carefully and use many interviewers per study.
Survey Administration Issues
- Postsurvey Error - Mistakes or artifacts from processing and analyzing the survey data. For instance, differences in coding open-ended responses. Consider using text analytics for coding large volumes of verbatim responses. Carefully review your survey analysis.
- Mode Effects - The effects introduced by the type of survey conducted. Human-administered surveys (telephone, face-to-face, videoconferencing) and self-administered surveys (IVR [Interactive Voice Response], paper, handheld, kiosk, online, SMS, etc.) have different strengths and weaknesses. Social desirability bias and satisficing are affected by mode. Choose the mode or mix of modes most appropriate to your research goals.
- Compatibility Effects - The difficulty of comparing surveys done at different times by different groups using different methods. Be cautious about contrasting findings from different types of surveys.
Where the focus in survey research has often been on minimizing sampling error, the focus should instead be on minimizing all types of survey error. The total survey error approach is a comprehensive method for doing so.