Site-Intercept Survey Best Practices
Posted by Jeffrey Henning on Tue, May 18, 2010
Several weeks ago I reviewed nine common applications of survey popups, for site feedback, e-commerce and general intercept surveys. What are the best practices you should follow to maximize the success of a popup survey?
Keep it short. Match the length to your research objective and the strength of your ties with your visitors. For shopping cart abandonment, asking one or two closed-ended questions and an open-ended question is about all you can expect people to complete. For site experience surveys conducted with regular customers, you can ask longer questionnaires, up to 10 minutes long, probing in more detail about what they were trying to accomplish at the site. Err on the side of short surveys.
Don't ask unless you don't know. While the survey should appear short, it should certainly embed information behind the scenes:
- Integrate with the shopping cart to pass in data about the transaction.
- Store browser and other session information.
- Determine site hygiene factors ("did the page load quickly?") from the site itself and from browser details.
- For sites where the visitor is a registered user, track the username so that you can import account information (but, if you do so, set privacy and confidentiality expectations appropriately).
Understand visitors before you ask. While many questionnaires are clearly written without an adequate understanding of the audience being surveyed, this can be particularly jarring on popup surveys. Assuming someone has visited your site directly and asking them detailed questions about how they navigated to the page is inappropriate if they just arrived via search engine. Make sure you really understand what they are up to before you write your questionnaire.
Paginate appropriately. Completing additional pages is an added burden for respondents, but inviting people to take a 20-question survey with all the questions on one page spikes the abandonment rate. Strike a careful balance, and paginate at logical breaks for skip patterns rather than writing out skip instructions.
Brand the survey. In an age of popup spam and phishing attempts, match your intercept surveys visually to your overall site to reassure visitors that this survey is affiliated with your site.
Intercept at the appropriate time. Listen to the complaint of this site visitor: "When I land at the site I want to proceed to whatever I'm there for. I don't want some annoyance in my face right away blocking me from accomplishing my goal. I'm more likely to take the survey if the request comes towards the end of the order/account process or after I've viewed some content." There is no optimal time to intercept: it's a tradeoff. You can invite visitors to hide the invite window and rate the site afterwards, but now you are changing their behavior and making them more self-conscious in their use of the site (which has pros and cons). Are you looking for feedback from people who have already spent some time on the site? If you have a 50% bounce rate per page, and you are doing a popup when the fourth page loads, you will only be interviewing 12.5% (50%^3) of your visitors. Don't interrupt people during key processes: surveying people as they put items in your shopping cart will likely result in a greater number of lost sales than if you didn't interrupt the flow. Perhaps you will get a higher response rate simply by asking them if you can email them a survey: when they are in the middle of trying to get their printer to work by downloading a driver from a website, they are not exactly in the mood to rate the site itself. An email a day later might be much more receptively received.
Invite a subset of visitors to the site to the survey, rather than all visitors. Site feedback is such a great idea, why not show the popup to everyone? Because it makes the results unrepresentative. The second key element of random sampling (after ensuring a chance that everyone can be selected) is external selection - that you invite someone to take a survey, rather than simply give everyone the option. Giving everyone the option means that many people will not take it: you have a convenience sample, which is much less likely to be representative of site visits.
Ask them if you can survey them again. Prompt them for their email address and invite them to join a panel of site visitors, which you can use for qualitative research into particular site-design issues as they arise later.
Test your technology. Many pop-up surveys aren't delivered because of browser settings; it's an arms race between advertisers and browser makers. Test your system on a range of browsers to make sure it works as intended.
What best practices have you seen?