Welcome to the Listening Post!

Sequential vs. Grouped Placement of Filter Questions

Tweet

 

At the 2010 AAPOR annual conference, Lisa Carley-Baxter of RTI presented a paper, "Effect of Questionnaire Structure on Nonresponse and Measurement Error: Sequential vs. Grouped Placement of Filter Questions" (by Lisa Carley-Baxter, Andy Peytchev and Michele C. Black).

The sequential placement of filters (also known as an interleaved format) involves asking a series of filtering questions (e.g., "Have you traveled internationally in the past 30 days?") with each followed by skip patterns to or around detailed probing questions depending on the answer to the filter. Sequential placement follows the pattern: filter, follow-up, filter, follow-up, filter, follow-up. Grouped placement follows the pattern: filter, filter, filter, follow-up, follow-up, follow-up.

The fear when using the sequential placement of a series of filters and follow-up questions is that respondents will recognize the pattern and realize that a certain type of answer results in them being asked more questions. The theory is that respondents will therefore lie when answering subsequent filters to get out of being asked more questions, leading to the underreporting of experiences.

Since structure can impact estimates through multiple types of measurement error, "researchers aim to structure questionnaires in the way that respondent's memories are organized". To test the impact of each structure, RTI used an experimental design of an RDD (Random Digit Dialed) survey with a split-test ballot. The survey, known as the NISVSS (the National Intimate Partner and Sexual Violence Surveillance System), asked about the number of abuses experienced by respondents from romantic partners. The filter questions in the RTI study are atypical. Instead of being a yes/no question, respondents were asked for a number which was then followed up on if non-zero: e.g., "How many of your romantic or sexual partners have ever slapped you?" The survey had a response rate (RR1) of 12.9% with 495 completes for the sequential structure and 289 completes for the grouped structure.

The good news was that RTI found no meaningful differences in the number of lifetime abuses reported, regardless of the format. The bad news was a regression model showed an interaction effect that varied by gender for the form used: high reporting for females (who may have preferred the flow and recall of sequential filtering) with low reporting for males (who may have been avoiding answering additional questions).

A qualitative review of the survey instrument with the interviewers found that both they and the respondents disliked the grouped structure. Respondents found it confusing to have to go back and consider follow-up questions about topics they had already discussed. Interviewers reported that the sequential structure was easier to administer.

My recommendations: Respect the respondent, by assuming the best of them and asking the questions in a more conversational flow, using sequential filtering. If you are very concerned about respondents falsely answering later filters (so that they can avoid the follow-up questions), then set up the filter/follow-up pairs in randomized blocks so that each follow-up section suffers equally any ill effects.