The Usability of Web Surveys
Posted by Vovici Blog on Wed, Mar 02, 2011
Paper surveys are sometimes tested for clarity and ease of understanding, but online surveys have opened up a whole new field of study: how respondents interact with the user interface of the survey. The usability of a particular survey can affect the experience in ways that are not always recognized.
In the February issue of AAPOR’s online journal, Survey Practice, Mick Couper of the University of Michigan and Reg Baker and Joanne Mechling of Market Strategies International share their paper, “Placement and Design of Navigation Buttons in Web Surveys”. They tested different configurations of the Next and Previous buttons (and, for one test, a Previous hyperlink) to determine the effect on use of the Previous function. It’s important to offer respondents the ability to change their past answers, while hoping they use it rarely. Counterintuitively, placing the Previous button before the Next button actually results in it being used less, resulting in marginally faster completion times.
Bernie Malinoff of element54, in his pioneering study, “Sexy Questions, Dangerous Results,” conducted eye-tracking research on survey participants as they completed basic online surveys and “Web 2.0” surveys. In the video below, a respondent gets stuck in the answer validation of a matrix question, leading to answers of dubious accuracy:
James Sallows, vice president of EMEA client operations with Lightspeed Research, shared some results of his firm’s usability research at this year’s Online Research Methods conference (see Causes of Survey Incompletes: Why Panelists Say They Abandon Surveys for a fuller account of his talk).
- Backing up research done by element54, Lightspeed found from its own eye-tracking studies that respondents often look at the list of multiple-choice answers before reading the question itself, and will skip reading the question text if they can infer it from the list of choices. “We don’t read pages, we scan them: get rid of half the words!”
- The body language of respondents confronted with matrix questions is particularly poignant: “People physically recoil the moment a grid comes up. They sigh. Their heads drop.” Even relatively small grids can seem overwhelming.
- For one recurring study, Lightspeed redesigned the interface, removing 140 mouse movements and 70 mouse clicks thereby shortening the length of the exercise 25%. As a result, the incompletion rate dropped from 42% to 12%, and data quality improved.
- Echoing Bernie’s research, James said, “Do not use technology just for the sake of it! Style over substance turns off respondents. Always ensure that the respondent’s role is at the center of the design.”
Good survey researchers can’t stop at understanding sound research methods – they need to understand the ramifications of the design choices they make when creating online surveys, and they may need to test unusual approaches. Usability is not just in the eye of the beholder, or the mind of the respondent. It shows up in the quality of the data you collect.