What You See Isn't What They Get: Differences in Survey Appearance
Posted by Vovici Blog on Tue, Feb 01, 2011
At the 2011 Online Research Methods conference, Greg Ward of FlyResearch discussed “Assessing How Device Convergence is Affecting Market Research”. Once upon a time, it used to be so simple, according to Greg: respondents had a desktop computer, and a cell phone, and they took surveys on their computer. Then they bought a laptop and took surveys on it. Then they bought a smartphone and took surveys on it. Then they bought a tablet computer and took surveys on it. Heck, some of them even bought Kindle ebook readers and took surveys on those.
All this is a nightmare for authors of online surveys. “People who you think are on an ordinary computer may be mobile. People on ‘mobiles’ may be sitting at home doing the survey. The distinctions are blurring… and it will get worse!”
How much “worse”? Over 70 percent of the world population has a mobile phone; already 7 percent of the world’s population have accessed the Internet from a mobile device. In many markets, usage of the mobile web will surpass usage of the traditional web in the next few years.
The vast range of hardware devices capable of accessing the web mean that is impossible for anyone to test their survey on all possible devices that respondents might be using to take the survey. The good news is that each device tries to offer broad support for Internet standards, to offer their own users the best experience possible. But where the devices vary widely is in the resolution of the screens they are using to surf the Internet. Don’t confuse resolution for physical size: the iPhone 4 has high resolution (640 x 960) but on a small screen.
Greg shared the following diagram, illustrating the range of resolutions available on different devices:
Nor is resolution the only difference between how devices render web pages. The 134 million iOS devices (74 million iPhones, 45 million iPod Touches, 15 million iPads) don’t support Flash survey elements. The good news is that such elements simply fail to work. Interactive elements programmed in JavaScript may work differently on different devices, with unpredictable results. Considering which, Greg asked, “Are you sure that fancy slider question adds anything?”
Not only do you not know what device respondents are using, but those devices can lie to you. It’s easy to configure a web browser to return a different User Agent String (captured by Vovici and most other survey applications) than the default setting. Tools like BrowserHawk To Go can help you inspect User Agent Strings.
Greg’s recommendations;
- Test your survey on multiple devices, then test again
- Analyze what devices respondents are using
- Keep surveys short, breaking up studies into multiple surveys over time
- Don’t stray beyond “boring” HTML without good reason
- Consider re-sizing surveys to suit different devices, while preserving wording
- Design some surveys specifically for mobile phones for appropriate research, such as “of the moment” and location-based studies
Summing up, Greg said, “Do not assume what you see is what all respondents see (it’s not like watching TV). There are hundreds of ways of accessing surveys—and lots more on the way!”