Best Practices in Mobile Research

Chris Ferneyhough and Sonia Bishop of Vision Critical discussed best practices for fielding online surveys to mobile audiences in the ESOMAR Online Research 2009 conference. Chris began by pointing out that mobile Internet adoption outpaces desktop Internet adoption and forecasts that eventually usage of the mobile web will be 10 times usage of the traditional web.

Vision Critical asked respondents to an online survey if they had received their email invitation on their phone: 1.9% had in the US, 1.2% in UK and 3.8% in Canada. Clearly, respondents are already completing online surveys on mobile devices, even though authors have often not taken this into account. Chris mentioned that a panel registration survey for a smart phone vendor didn't actually work for that smart phone, because of the registration form's reliance on JavaScript, which was off by default on the phone: the open-ended "Other (please specify)" box was locked and disabled because it required JavaScript to enable it once the corresponding radio button was clicked. Since client-side scripting is disabled on many phones, the data your survey collected may be wrong.

Researchers need to recognize the fact that online surveys are being completed on mobile devices and need to be optimized for that medium. The wide variety of smartphones at GSMArena.com reveals hundreds of different models with dramatically different market share in different countries. Colors and fonts are implemented differently by different phones and may not be implemented at all on a smart phone: a certain color may render some text unreadable.

Many respondents are unfamiliar with their web browser or alphanumeric entry mode on their phone's keypad. Many respondents are concerned about data costs: some have unlimited data plans, others have pay-as-you-go plans. Many have low data connection speeds.

To do further research on these topics, Vision Critical studied respondents who are smartphone users and are willing to complete questionnaires on their phone. The sample was balanced and weighted on gender and age, with 500 Canadian respondents, 118 US respondents and 107 UK respondents. The survey covered attitudes towards the national economy.  No significant statistical differences were found for closed-ended questions on mobile devices vs. desktop devices. For open-ended questions, of course, desktop users were more verbose.

The likelihood to participate in future surveys on mobile phones was greatest for iPhone users (47% were likely to), compared to only 34% of Blackberry users and 23% of all other smartphone users.

Sonia presented back-end mobile research best practices:

• Maximize use of the available space
• Profile your panel for smartphones, whose email addresses may vary for the phone vs. the desktop
• Identify devices and models supported by your data collection software
• Manage the process of deploying surveys to mobile panelists

And questionnaire design best practices:

• Use simpler question types
• Avoid Flash effects and JavaScript validation
• Write more concise questions and answer lists to minimize scrolling
• Put the Next button "above the fold"
• Limit survey length to 10-15 questions (unless heavily incentivized)
• Keep it simple: avoid color, grids, images, etc.
• Develop for the lowest-common denominator devices

Clearly, survey authors fielding online surveys need to take mobile users into account when developing surveys.

ESOMAR 26 Q and Internet Guidelines

Kees de Jong, the CEO of SSI, presented an overview of ESOMAR codes and guidelines at the 2009 ESOMAR Online Research conference.

ESOMAR is less well known in the United States then the rest of the world; ESOMAR is an international organization promoting better market research, with 5000 members in 100 countries agreeing to abide by its code. The United States found online access panels to be of acceptable quality for years, only refocusing on quality after Dedeker's criticisms of quality at the IIR in 2006. This led to a dramatic surge in interest in quality, with many initiatives from many of the industry associations.

The ESOMAR Code outlines eight key fundamental principles for professional researchers. The Code dates back to 1948 and first addressed Internet research in 1997, with the Internet guidelines having been revised three times since then, most recently in the upcoming 2010 guidelines. The E25/E26 provided a list of questions to help inexperienced buyers of online panel compare and select providers. Providers then asked ESOMAR to come up with the right answers, but the goal was to stimulate debate not provide authoritative answers, which was healthy, as new research changed assumptions: for instance, NOPVO discovered that surveys with low response rate (5%) were not materially different from surveys with high response rate (30+%).

The 2010 guidelines look at four focal points: ethical (e.g., confidentiality and anonymity), regulatory (e.g., PII), methodology (e.g., panel vs. river), and technology issues (e.g., digital fingerprinting). The three overriding principles: researchers must diligently maintain distinction between marketing and research; don't bring the industry in disrepute; and the foremost principle is treating the respondent with respect, building a relationship built on trust, respect and reciprocity.

What does the future hold? A new focus on respondent experience and perhaps an ESOMAR Experience Conference!

Social Media Trends around the World

Bonnie Breslauer of Lightspeed Research presented the Global Web Index (GWI) at ESOMAR Online Research 2009, which includes research from Tom Smith of TrendStream.

The Global Web Index is a syndicated research offering from TrendStream focusing on attitudes regarding usage of social media. TrendStream surveys 16,000 respondents every six months with a long questionnaire (completion took longer than 30 minutes). Lightspeed's global consumer panels were weighted by national representation of key demographics, focusing on 16 markets representing 70% of the Global Web Universe.

While email is continuing to grow, social network and community sites are growing faster: 31% growth from August 2008 to 2009 for social networks compared to 20% email growth. The key factors for online usage are cultural, generational and gender.

• Cultural drivers. Emerging markets (Brazil, Russia, India, China) have the greatest usage of social media and use social media more than ecommerce.
• Generational drives. After country, age is the second defining factor. The younger the respondent the more likely to use social media. Everyone uses the Internet to stay in touch with friends, but keeping friends up-to-date has the highest skew towards younger generations. Users who create content are also more likely to be younger, regardless of culture. Younger generations have greater trust in social-media connections and trust the author of a blog they've read recently more than a journalist for a national periodical.
• Gender. Women are more socially motivated than men and are more likely to have engaged in social media usage within the prior month.

Social media users fall into three types: passive, light and active. Passive users participate by viewing; light users update content or manage a social-network profile; active users create web sites, blogs or videos. While small in number, active users have a disproportionate influence.

The survey touched on some brand use and perceptions. Consumers value open dialogue with a company as opposed to one-way messaging: 22% reported that their perception of a brand is improved if the organization responds to comments in online communities and forums. In an interesting segmentation, owners of Chrysler, GM and Ford automobiles are less likely to use microblogs or upload videos than BMW or Mini Cooper owners.

As social media evolves, it is more important than ever to understand trends among consumer usage of social media.

E-mail List Rental Guidelines

E-mail list rental from reputable third-parties is one avenue for obtaining e-mail addresses to invite to online surveys. When using a third-party list, keep these four guidelines in mind:

1. Reputable providers will only rent the e-mail list to you: they will never sell you a list. If a list owner offers to sell you the full contact names and e-mail addresses of their list, you should strongly avoid using this type of list since it is most likely a non-permission-based list and could put you at risk for violating the CAN-SPAM Act.
2. Reputable providers will send your content out to their list from their e-mail system. You will not have the ability to review the names and their contact information nor will you have the opportunity to obtain a copy of the list. You'll only obtain the responses to your survey.
3. Reputable providers will only rent double opt-in lists. Double opt-in lists are created when an individual registers on a website and provides permission to contact them via e-mail. Once they have opted-in to the list, a confirmation e-mail is then sent to the e-mail address provided in their initial registration. Only by clicking on a link confirming that they are providing their permission for communications will they be added to the list. By requiring a second level of confirmation to the registered contact, list owners are assured that no one will be added to the list without the permission of the contact.
4. You pay for one-time use of the list regardless of the response rate to your survey. When you rent a list, all responsibility for obtaining responses is yours, and you must make sure that you have a compelling invitation and incentive in order to justify the expenditure.

Please keep in mind that most research conducted using rented email lists is qualitative; because respondents were not selected randomly from a target population, you cannot extrapolate to a wider audience. The exception is when a magazine or website itself uses its own list to measure attitudes among its subscriber base; such surveys are typically projectable to the audience of readers. For some research, especially into low-incidence populations, qualitative studies using email lists are simply the only cost-effective approach.

'No Opinion' as a Question Choice

 

According to many seasoned survey researchers, offering a no-opinion option should reduce the pressure to give substantive responses felt by respondents who have no true opinions. By contrast, the survey satisficing perspective suggests that no-opinion options may discourage some respondents from doing the cognitive work necessary to report the true opinions they do have. We address these arguments using data from nine experiments carried out in three household surveys. Attraction to no-opinion options was found to be greatest:

1. among respondents lowest in cognitive skills (as measured by educational attainment),
2. among respondents answering secretly instead of orally, 
3. for questions asked later in a survey, 
4. and among respondents who devoted little effort to the reporting process. 

The quality of attitude reports obtained (as measured by over-time consistency and responsiveness to a question manipulation) was not compromised by the omission of no-opinion options. These results suggest that inclusion of no-opinion options in attitude measures may not enhance data quality and instead may preclude measurement of some meaningful opinions.

 

Paper Survey to Web Survey

Our customer base strongly skews to an older, retired (and presumably less technically-savvy) demographic. Accordingly, we have continued to use traditional postal surveys to measure satisfaction in this conservative customer base.

 

We are eager to switch to an online survey methodology, but have two related concerns: (1) comparability of historical data (data collected in postal survey vs. online) and (2) switching methodology may effectively 'disenfranchise' some older customers who may not have email access.

 

From a technical standpoint, how serious are these concerns and what, if anything, should we do to address these concerns -- either in terms of survey methdology or at the analytical stage?

Skip Logic & Conditional Branches in Surveys

Survey respondents can be routed through questionnaires using three types of branching logic:

• Skipping specific questions: A skip pattern will jump a respondent over a group of questions that isn't relevant to them. For instance, a common skip pattern has a respondent rate a particular attribute on a scale and, if the rating is high, skips the respondent over one or two follow-up questions designed to probe why other respondents gave the item a low rating.
• Conditional branching: A branch pattern will route a respondent to the appropriate section of the questionnaire: each respondent follows one of the branches. A common branch pattern has a respondent classify a product purchase or a type of service interaction and then asks follow-up questions specific to that answer.
• Unconditional destination: Often a branching pattern is terminated with an unconditional destination, which returns respondents back to a common path. This has the effect of jumping respondents over other branches.

When you find yourself writing a question that starts with something like "If you answered ‘No' to the previous question", that is a sure sign that you should set up a skip or branch pattern. Making people do this logic in their head is fine in a paper survey, but it is in appropriate for online surveys. Such manual skip patterns slow down the respondent, increase the amount of reading that they must do and make completing the survey tedious. Conversely, when skip and branching patterns are well implemented, they make the survey highly relevant and engaging.
A great example of well-implemented branching is a web-site feedback survey created by one Vovici client. With over 15 million unique visitors a month to their web site, the client determined that visitors typically came to the site with one of 17 different purposes in mind. Visitors to the site were randomly invited to take the feedback survey, which began with five standard questions that all respondents were asked, ending with a question asking "So what are you here to do today?" The answer to this gating question would then branch respondents to one of 17 different paths, each asking an average of 8 questions unique to that action. The result was that no respondent answered more than 20 questions of a 148-question survey, and the questions they did answer were very specific to their experience.
Your own skip and branching patterns don't need to be as complex. Some best practices for skip and branching logic:

• Remember that page breaks indicate logical jumping-off points. Sometimes survey authors like to have one question per page; other authors like to have related questions together on a page. Skip logic overrides the preference for how to group questions, as it is only applied at the end of a page.
• Skip logic routes respondents forward through the survey, never backward. Think of the question flow like a river, going downstream, with different sluice gates channeling the water into different canals and sometimes back into the main watercourse.
• Due to the wide variety of possible synonymous answers to open-ended questions, skip logic is primarily used with closed-ended questions.
• Before inviting respondents to take your survey, make sure you test that each path through the survey matches the logic you intended.
• Some survey software applications, include Vovici v4, support Boolean logic ("and", "or" and "not" and nested parentheses). This makes programmers happy, but can be confusing for first-time survey authors. For instance, "and" doesn't have its English meaning: a survey author who wants the system to skip to Q9 when Q1 is answered "blue" and "green" needs to write "Q1=blue or Q1=green" not "Q1=blue and Q1=green" (which is a logical impossibility from a Boolean perspective). If you are using advanced branching for the first time, get some help from technical support or a programmer and test your survey logic carefully.
• For a questionnaire already collecting responses, you sometimes might find that you should have added a skip pattern. Good survey software will let you do this, then provide a utility to validate all previous answers against this new skip pattern, setting any answers back to unanswered for questions that shouldn't have been asked.
• If you are displaying progress bars, respondents may suddenly find themselves very far through the survey, because of a skip pattern. Curious respondents may hit the Back button, then answer differently to see where they go next. Respondents are more likely to do this in the screener if the survey offers a financial incentive to qualifying respondents. In such cases, configure the survey to not publish a Back button beside the Next button.

What other best practices do you follow when using skip logic?

Less Really is More When it Comes to Response Scales

Two years ago, Ziggy Zubric, owner of Marketing Endeavors, and I engaged in a passionate debate about a survey we were collaborating on. Ziggy made the case for replacing our traditional five-point scales with something more innovative. I decided his approach wouldn't work for the task at hand, but his enthusiasm and unique point of view stuck with me. Recently, when writing a questionnaire, I recalled his advice and decided to invite him to write a guest post about his work.  - Jeffrey

Every researcher (and every client!) should be interested in developing surveys that:

1. Look quick and easy to take
2. Generate maximal variance (i.e., good distribution among all response options)
3. Avoid "response set" (i.e., respondents mindlessly giving the same answer for every question)

After of years of conducting online surveys, we've developed some strong opinions on how best to do this, and it starts with (a) replacing traditional five-point scale items with three-point scales, then (b) using a non-traditional answer order.

For context, let me state that we focus heavily on the in-store customer experience. That is, after customers have visited a bank, restaurant, grocery store, retail outlet, etc., we want to discover their thoughts and feelings about the experience. 

Importantly, we do not believe in asking customers about specific details. For instance, unless a client demands it, we never ask customers to recall whether eye contact was made, if employees were wearing name badges, etc. Customers rarely retain accurate memories of such issues (why should they?), and if asked such questions, they simply guess.  

Thus, the most you can reliably do in a post-experience customer survey is to explore overall perceptions. Customers can't accurately tell you if a teller introduced herself, smiled, and expressed appreciation for their business, but they can report if the teller created an impression of professionalism, competence, compassion, patience, etc. (They can also accurately share their feelings on satisfaction, loyalty, brand perceptions, etc.) 

Benefit #1- It looks easier to take

First, our experience shows that respondents are far more willing to take and finish surveys that are visually simple and appear to require little effort. This leads to less fatigue and far greater completion rates. 

And let's face it, using only three answer options takes up a lot less visual space and is far less daunting than five options.
Thus, we always try to limit the response options to three, as that number offers the robustness to capture what we need while still remaining visually inviting.

Benefit #2- It generates more variance

One of the first objections people usually raise is that limiting respondent options to only three answers greatly reduces variance. But our experience shows that exactly the opposite occurs. 

Years of data show that the spread of a five-point scale usually skews heavily towards the top two answer options. Thus, when using five-point scales, we'll find that:

• 88% of respondents answer 4 or 5
• 10% answer 3
• 2% answer 1 or 2

We see this pattern whether using numbers (e.g., 1 through 5) or any variation of verbal descriptors (e.g., Very good, Above average, Average, Below average, Poor). 

The bottom line is that variance can't get much worse than it already is. 

This can't be valid

How believable is it that 88% of all customer experiences are ‘very good' or ‘excellent'? Does that jive with your experience? I've asked a lot of people that question, and not one of them has ever said "Yes."

The reality is that service experience data collected with five-point scales greatly exaggerates the quality of service. And practically, telling customers that their average score on a five-point scale is between 4.6 and 4.8 for every item simply isn't helpful. 

What clients ultimately want to know is: When are we so good/bad that the experience is memorable and noteworthy, and how often are we extremely good/bad?

According to five-point scale data, 88% of service experiences are Good or Excellent. But we all know that's not true. The majority of service experiences are average. 

But occasionally customer service is extremely wonderful or remarkably horrid, and those are the experiences we want to make sure we capture.

With five-point scales, a typical, efficient, mildly positive experience will be rated as a 4 by some respondents, and a 5 by others. And a negative experience will be rated as a 4 by some respondents, and a 3 by others. In effect, our categories stop being mutually exclusive, vitiating the validity of the data.  Ultimately, what we want to accomplish is to differentiate truly positive/negative experiences from the vast majority of average, just-fine experiences, and a five-point scale just doesn't get us there.

So why do respondents provide such glowing reviews? 

That requires a lot more discussion, but we're dealing with at least three psychological tendencies. 

First, respondents feel a sense of guilt about giving scores that are very low, so they avoid the bottom categories.

Next, respondents are far too willing to describe an average, typical, yet perfectly acceptable service experience as Excellent or Very good. 

Finally, respondents who are simply trying to get through the survey (aka, response setters) will mindlessly click the top answer option every time, creating higher positive scores. 

The three-pointer

Let's deal with overcoming the first two psychological tendencies. Basically, they both result in respondents being far too generous in their assessments. When we employ a three-point scale, we can word the response options to help customers provide more accurate feedback. 

Thus, our most negative option needs to sound very mild, making it less guilt-inducing. Our middle option needs to sound very positive, making it more acceptable as an option. Finally, our positive option needs to sound so stupendously, impossibly, incredibly tremendous that respondents understand to reserve it only for truly excellent experiences. 

Some wording examples we've used when assessing customer service:

• Above and beyond
• Very good
• Could use some tweaks

Or:

• Truly outstanding
• Perfectly acceptable
• Not the best

Clients that we have migrated to this system show more variance in their data, and the response options overlap far less, which increases validity. Now, the strong majority of responses fall into the middle category, which is just what we want.

And when someone checks "Truly outstanding," we are far more confident that our client's staff genuinely did provide extraordinary care. Similarly, when we see a "Not the best," we know that something was truly amiss. 

Also-and this might make traditionalists uncomfortable-don't be afraid to play with the wording. Different clients and different contexts require nuances. As long as you're consistent within that client (or within that industry if you're providing benchmark data), you'll be fine. 

Not to mention the fact that respondents are sick to death of traditional wording options, and they find unexpected answer options refreshing.

Benefit #3- Avoid response set

Sometimes respondents will put their brain on cruise control while taking a survey, and simply click on the same option (often the top one) for every question. To overcome this, list the "middle" option first. For instance, try:

• Very good
• Could use some tweaks
• Truly outstanding

Does this violate traditional ordering of categories? Yes. But so what? 

By listing your middle, unremarkable option first, people just mindlessly clicking the top option won't give you a false sense of good or bad news; and the respondents who truly have an extremely good or bad experience to relate will look for their appropriate response and check it, providing us more certainty that these really were exceptional experiences (for better or worse).

So overall ...

Vovici Online Survey Software

- Ziggy Zubric, owner, Marketing Endeavors

Checkboxes vs. Radio Buttons in Web Surveys

A survey from one of our customers was recently lampooned in a popular blog about “curious perversions in Information Technology”.

Clearly, the author of the questionnaire didn’t intend to set the validation for question 8 to limit to one choice, given their instructions to the respondent (“please tick all that apply”).  This was simply an honest mistake on their part.

But why should our survey software even allow the author to limit a choose-all-that-apply question to one choice?  If only one choice is permitted, then the system should use radio buttons or a dropdown box.  Early on, in fact, I wanted to make sure that this wasn’t even an option for questionnaire authors, but our customers pushed back that they wanted this functionality, for two reasons:

1. Some users don’t value the difference between radio buttons (shown in question 9, limited to one choice) and checkboxes, a distinction in user interfaces that dates back to at least 1984.  These users like the visual esthetic of the checkboxes better and choose to use checkboxes on all questions, even those where only one choice is permitted.
2. Some users dislike the fact that you cannot unselect a radio button, and prefer to use checkboxes so that a respondent can uncheck a choice if they decide no choice is applicable.

So, while you can certainly use checkboxes in this way, the best practice should be to use radio buttons when you only want the respondent to select a single choice, and to provide a “Not applicable” or “Does not apply” choice on radio-button questions when you want respondents to be able to “unselect” a choice.

Survey Software vs. Enterprise Feedback Management

How do survey software applications typically differ from enterprise feedback management solutions? 

Survey software is typically designed for the individual user, working on one or more survey research projects.  Core functionality includes:

• Survey Creation - Writing multi-page web surveys with open-ended and closed-ended questions.
• Templates - Starting with a wide selection of customer satisfaction, employee satisfaction, course evaluation and other standard survey instruments to choose from.
• Basic Conditional Logic - Skipping over certain questions and pages based on the answers to other questions.
• Multi-Language Support - Fielding one survey in multiple languages.
• E-mail Invitations - Sending out invites with hyperlinks to potential survey respondents and following up with periodic reminders.
• Basic Reporting - Viewing a frequency report or a list of verbatim responses for each question.
• Exportable Data - Saving survey data to file formats such as Excel and text files.

Vovici knows survey software well.  The PC Magazine roundup of six survey software applications in February 2000 featured three products - Perseus SurveySolutions for the Web, Websurveyor 2.0 and EZSurvey 99 for the Internet -  that are all part of Vovici's heritage (Vovici was formed from the merger of Websurveyor and Perseus Development, and purchased the Raosoft EZSurvey assets).  While Vovici knows survey software well, we've made a business of supporting customers as they've outgrown the needs for such software.  Vovici pioneered enterprise feedback because so many survey software customers needed more functionality.

Today, enterprise feedback management systems typically extend the core functionality of survey software with:

• CRM Integration - Where users grow tired of exporting email lists from their CRM system, CRM connectors can keep the EFM system in sync automatically.
• Advanced Analytics - When users want to drill down on results in detail, EFM survey analytics go beyond  simple reports.
• Triggers & Alerts - When users begin to think about survey processes, rather than projects, they can set up email alerts triggered by specific answers to the survey.
• Report Distribution - When users want to share results widely across an organization, or customize reports for different audiences or even for each individual, EFM report distribution can save time and energy.
• Quotas - When users want to limit participation in surveys, for purposes of offering incentives or for balancing responses by key segments, they can easily set survey quotas.
• Panel Management - When customers or other key constituencies are complaining about receiving too many surveys, panel management can be used to set up limits on frequency of surveys.
• Page Rotation - When respondents complain about surveys that are too long, page rotation is one tool for shortening the questionnaire from the respondents' perspective.
• User Management - A key part of sharing usage of surveys throughout an organization is defining different roles for different types of users:  some can write questionnaires, some can create reports, some can only view reports, and so on, as appropriate.
• Workflow Management - Users can draft surveys and then submit them to reviewers to edit, approve and publish, thereby by ensuring that research best practices are followed.
• Online Communities - In the past year, EFM systems have expanded to support online communities, providing better qualitative research than ever before.
• Advanced Branching - As questionnaires grow in sophistication, authors often add more complex rules around which questions are shown to respondents, so that they are answering only the most relevant questions and do not have to manually skip over questions that aren't appropriate to them.
• Response Randomization - Finally, to remove any order bias, the choices for questions can be shown in random order.

If your organization has any of the above needs, it is time to consider moving up from survey software to enterprise feedback management.