Welcome to the Listening Post!

Bullet Time for Matrix Questions

Tweet

 

Among many other things, the movie The Matrix is known for its popularization of bullet time®, a special effect where time is slowed so that you can actually watch bullets move through space and see the characters dodge them. When a survey respondent tries to answer a matrix question (if they don't dodge them), time seems to slow as well, as they work through tedious row after tedious row; they actually answer the questions quickly, but it doesn't feel that way.

Matrix questions are compact and visually efficient, but they suffer from two primary problems.

1. A long record of research on research shows that matrix questions increase satisficing and survey abandonment (Krosnick, Narayan & Smith, 1996; Comley, 2000; Schonlau, Fricker, Elliott, 2002; etc.). A prominent type of satisficing behavior in matrices is known as straightlining, selecting the same column in row after row; this requires less mental and physical effort than choosing the most appropriate answer for each row. Many respondents will select a column, check the radio button in most rows in that column, varying to the left or right only for a few items. The result is a reduced range of differentiation between answers.
2. Earlier research that seemed to demonstrate that matrixes offer higher data quality (Couper, Traugott, Lamias, 2001) has largely been superseded by reports showing a loss of quality. Wide rows can visually confuse respondents about which item they are rating, as can the presence of too many columns (Gräf 2002). "We know respondents don't like grids," said Jackie Lorch, SSI VP, in a March 2009 press release. "They've been telling us that for years in focus groups and feedback, but we've always thought of grids as a necessary evil in questionnaire design. Now, we're beginning to learn that not only are grids frustrating for respondents - they actually produce inferior data." The SSI study found that a matrix refactored into many separate questions produced results with greater cohesiveness (.931 Cronbach's alpha compared to .799) and higher predictive validity (13.67 X2 compared to 3.75).

How bad are matrices? The blogger known by the alias ResearchRants collects bad examples in a category entitled "matrixes make me cry". Robert Moran of StrategyOne, in his Survey Research Resolutions for 2010, resolved to "Avoid the so-called ‘death grids' [large matrix questions] in online surveys." In an earlier post, he said, "No normal human has the patience to fill out a ‘deathgrid' and no non-incarcerated individual should be forced to do it."

In "Satisficing in surveys: Initial evidence", Krosnick, Narayan & Smith argue that you should re-factor matrix questions into simpler questions, an approach validated by the SSI study. Here are some possible refactoring strategies. Keep in mind that each of these will change the data quality and will introduce complications when comparing the new data collected to past results collected through matrices.

• Break each row of the matrix into a separate question or group of questions on its own page. Rewrite each row using the question types (e.g., choose one, choose many, open ended) as specified by the columns. In other words, the questions are the same but have been unpacked from the grid.
• Rewrite each row of the matrix into separate questions, replacing checkboxes with fully labeled scales. This was the approach favored by SSI, which produced high data quality but at the cost of taking twice as long to complete.
• Refactor importance matrixes into choose-many questions. This reverses the SSI approach, collapsing scales: For instance, rather than ask respondents to rate a list of attributes on an importance scale, ask them to select the most important items. While respondents are now only distinguishing between two classes of importance (important/checked and unimportant/unchecked) instead of five, in aggregate you will see wider differences between attributes than with the matrix question, and you will eliminate the possibility of straightlining. Respondents are now actually trading off which items are more important than others, an exercise they can avoid when filling in a matrix.
• Refactor Yes/No Matrix questions into choose-many questions (checkbox lists) instead. The respondent will not have to click a response for each item, resulting in less effort. Unfortunately, the two question types are not exactly equivalent: fewer boxes will be checked than radio buttons will be set to "Yes", according to Dillman, Smyth, Christian and Stern (2006).
• Be creative. Think about how you are going to analyze the data and see if other question types and skip patterns can produce similarly useful information with less effort from respondents.