SharePoint as Survey Software

We often have prospects looking to migrate to a survey software application from a home-grown system. Recently, we've been seeing more organizations looking to move from Microsoft SharePoint, an enterprise content management platform.

What clients liked about SharePoint when it comes to building online surveys:

• The WYSIWYG form editor makes it easy to build basic surveys.
• It offers complete flexibility for programmers to use HTML, JavaScript, AJAX techniques, Flash and other standard web development approaches.
• It stores data directly in the repository that the administrator selects (e.g., SQL Server or Oracle).
• Survey results are readily exported as CSV (Comma-Separated Value) or XLS files.

Here are the features that caused SharePoint users to move to a dedicated feedback platform:

• It is difficult for business users to create surveys of moderate or significant complexity with SharePoint.
• The lack of integrated email support meant that invitations, reminders and thank-you emails had to be created outside the system.
• The lack of panel management meant administrators couldn't target respondents and track past survey activity.
• It is difficult to personalize surveys based on known data.
• SharePoint lacks survey-specific web services, making integration more tedious.
• Some users reported security concerns.
• Until recently, there was no support for basic skip patterns (available with MOSS 2007) or advanced branching.
• It has no real-time reports or dashboards and can't export results to SAV or PPT files.

SharePoint is a great interim solution for organizations first adopting web surveys. It excels at short, uncomplicated and static surveys, but organizations outgrow it when they need advanced questionnaires, intermediate MR functionality, integrated email invitations, panel management or enterprise feedback management.

Best Practices in Mobile Research

Chris Ferneyhough and Sonia Bishop of Vision Critical discussed best practices for fielding online surveys to mobile audiences in the ESOMAR Online Research 2009 conference. Chris began by pointing out that mobile Internet adoption outpaces desktop Internet adoption and forecasts that eventually usage of the mobile web will be 10 times usage of the traditional web.

Vision Critical asked respondents to an online survey if they had received their email invitation on their phone: 1.9% had in the US, 1.2% in UK and 3.8% in Canada. Clearly, respondents are already completing online surveys on mobile devices, even though authors have often not taken this into account. Chris mentioned that a panel registration survey for a smart phone vendor didn't actually work for that smart phone, because of the registration form's reliance on JavaScript, which was off by default on the phone: the open-ended "Other (please specify)" box was locked and disabled because it required JavaScript to enable it once the corresponding radio button was clicked. Since client-side scripting is disabled on many phones, the data your survey collected may be wrong.

Researchers need to recognize the fact that online surveys are being completed on mobile devices and need to be optimized for that medium. The wide variety of smartphones at GSMArena.com reveals hundreds of different models with dramatically different market share in different countries. Colors and fonts are implemented differently by different phones and may not be implemented at all on a smart phone: a certain color may render some text unreadable.

Many respondents are unfamiliar with their web browser or alphanumeric entry mode on their phone's keypad. Many respondents are concerned about data costs: some have unlimited data plans, others have pay-as-you-go plans. Many have low data connection speeds.

To do further research on these topics, Vision Critical studied respondents who are smartphone users and are willing to complete questionnaires on their phone. The sample was balanced and weighted on gender and age, with 500 Canadian respondents, 118 US respondents and 107 UK respondents. The survey covered attitudes towards the national economy.  No significant statistical differences were found for closed-ended questions on mobile devices vs. desktop devices. For open-ended questions, of course, desktop users were more verbose.

The likelihood to participate in future surveys on mobile phones was greatest for iPhone users (47% were likely to), compared to only 34% of Blackberry users and 23% of all other smartphone users.

Sonia presented back-end mobile research best practices:

• Maximize use of the available space
• Profile your panel for smartphones, whose email addresses may vary for the phone vs. the desktop
• Identify devices and models supported by your data collection software
• Manage the process of deploying surveys to mobile panelists

And questionnaire design best practices:

• Use simpler question types
• Avoid Flash effects and JavaScript validation
• Write more concise questions and answer lists to minimize scrolling
• Put the Next button "above the fold"
• Limit survey length to 10-15 questions (unless heavily incentivized)
• Keep it simple: avoid color, grids, images, etc.
• Develop for the lowest-common denominator devices

Clearly, survey authors fielding online surveys need to take mobile users into account when developing surveys.

Survey Testing Case Study

Survey Test Mode

In a similar way, many users of our survey software dislike the fact that the first time they publish a new survey, it goes into test mode.  This is a not-so-subtle reminder that they should double-check the survey before inviting participants to respond.

Self Test

Pre-Test

That last question is particularly hard to answer in a self-test.  When I am feeling very unsure of a study, or the results are strategic rather than tactical, I will pre-test it on coworkers or, even better, on a small sample of the target audience (no more than 50).  I will end the survey with some questions about the survey itself, to identify areas or survey structures that were confusing or ambiguous. 

Pilot Test

Publish

Common Rating Scales to Use when Writing Questions

One of the most frequent mistakes I see when reviewing questionnaires are poorly written scales. Novice survey authors often create their own scale rather than using the appropriate common scale. It's hard to write a good scale; instead you are better off rewording your question slightly so that you can use one of the following.

Acceptability	Not at all acceptable, Slightly acceptable, Moderately acceptable, Very acceptable, Completely acceptable
Agreement	Completely disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Completely agree
Amount of Use	Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness	Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate
Awareness	Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware
Beliefs	Not at all true of what I believe, Slightly true of what I believe, Moderately true of what I believe, Very true of what I believe, Completely true of what I believe
Concern	Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned
Familiarity	Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar
Frequency	Never, Rarely, Sometimes, Often, Always
Importance	Not at all important, Slightly important, Moderately important, Very important, Extremely important
Influence	Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential
Likelihood	Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely
Priority	Not a priority, Low priority, Medium priority, High priority, Essential
Probability	Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable
Quality	Very poor, Poor, Fair, Good, Excellent
Reflect Me	Not at all true of me, Slightly true of me, Moderately true of me, Very true of me, Completely true of me
Satisfaction (bipolar)	Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied
Satisfaction (unipolar)	Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied

This list follows Krosnick's advice to use 5-point unipolar scales and 7-point bipolar scales.

Let me know any of your favorite scales that I omitted.

Hurdles in Race to Turn Recipients into Respondents

You've chosen the list of people you want to invite to your survey.  You still have a lot of hurdles to jump to get each recipient to actually take your survey:

• Make sure their email address is not on the unsubscribe list
• Keep your survey invitation from being flagged as spam
• Craft a survey subject line and From field that help get the invitation opened  
• Cover the six points important to respondents in an invitation
• Write an invitation that is compelling enough that recipient clicks on the link to take the survey

Representative Web Surveys Require Good Email Lists of Customers

Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.

Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don't, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.

Organizations with unrepresentative email lists need to take the following steps for web-survey success:

• Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
• Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
• Conduct a telephone study or paper survey of customers for which your organization doesn't have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
• Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication. 
• Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.

What steps have you taken to make sure that your web surveys are representative of your customer base?

Getting Your Survey Invitation Opened

Your survey invitation gets past the spam filters and shows up in someone's email inbox. Now the recipient will have two or three items at hand to decide whether to click Delete or to open your message:

• From field - The From field consists of the sending email address and the alias for that address. Nothing will get your email deleted faster than having an impersonal email address without an alias, such as - a particular favorite! - noreply@example.com. Always include the alias. What type of alias works best varies by organization; examples to experiment with for your invitations include:
  • The name of an individual - The name of an employee managing the survey or commissioning the survey can be very effective.
  • The name of the brand or company - While this can be impersonal, it may outperform the names of individuals for B2B vendors.
  • Brand name + "Surveys" - For some organizations a From field such as "Acme Surveys" works best.
  • Hybrid - Some panel companies test combinations such as "Acme Surveys & Jane Doe".
• Subject line - The text of the subject line will have the biggest impact: "Your Feedback Needed", "Some Quick Questions for You", "Help Us Serve You Better", "Learn What Your Peers Think About -----" are all subject lines that have been successful for survey invitations. Your organization will need to experiment to determine what works best for your audience.
• First line of the invitation - Some email programs display the first line of the invitation in grayed-out text immediately after a bold subject line. All the more reason to pay special attention to the copy you use in the body of your invitation.

What tactics have improved your organization's open rate on surveys?

Types of Questions: Four Building Blocks for Constructing Questionnaires

Sift out the finer distinctions and most questionnaires are constructed from four basic building blocks. These four, fundamental question types are:

Open-ended questions	Essay Questions	Long text responses
	Fill-in-the-Blank Questions	Short text responses
Closed-ended questions	Choose-One Questions	Single choice selected
	Choose-Many Questions	Multiple choices selected

Essay Question

The essay question, in a web survey, is displayed as a multi-line TEXTAREA: 

Respondents can enter a few words, a few paragraphs, and even a few pages. Typically essay questions store 32,000 to 64,000 characters of text. We've had respondents copy and paste dozens of pages of text into essay questions.

This question type is best used for understanding in detail what a respondent believes in their own words.

Fill-in-the-Blank Question

A fill-in-the-blank question is displayed as one or more text boxes with short labels, and is designed for gathering short responses:

It can also be used for contact information or address information:

Responses to text boxes are often validated to follow a common pattern.  Common validations include:

• Email address
• Whole number within a range
• Real number within a range
• Date
• Telephone number - U.S./Canadian format or international format
• Zip codes and postal codes

Choose-One Question

The most common question type is that of the single-select multiple-choice question, where a respondent chooses one and only one of the available options. Choose-one questions are typically shown with radio buttons or dropdown boxes.
While choose-one questions constrain the choices of the respondent, they are much quicker to answer and much easier to analyze.  

Choose-Many Questions

The multiple-select multiple-choice question allows the respondent to check all the choices that are applicable to the question.  Most survey software uses the standard checkboxes of HTML forms to show these questions:
These four questions can then be combined together in matrix questions.
For best practices for each type of question, see these posts:

• Essay-question best practices
• Choose-one question best practices
• Choose-many questions  

Compelling Survey Invitations

You've gotten the recipient to jump all the hurdles to get to the survey except for the last one - reading the invitation and clicking on the survey link.

You now have the challenge of writing a short but motivating invitation that will prompt your recipient to become a respondent.

As with any attempt at marketing, you have less than eight seconds to make a first impression. A clever use of images can help catch people's attention. You need to tell recipients what you want them to do as soon as possible in the invitation. Know your audience and write the invitation with this knowledge in mind. Keep the invitation short-but cover the key points.

Patrick Glaser's "A Playbook for Creating Survey Introductions for Online Panels" provides a useful outline for a survey invitation, along with example phrases.

I wish I could give you some proven catchphrases for subject lines that work with every survey, but what worked once--"Help us improve our products" or "Share your opinion"--might not work on the next round of survey invitations to the same audience. Experimentation and inventiveness are essential.

Do not make privacy claims that you may not keep. This tends to happen by mistake rather than from any intention to mislead recipients. Often you are surveying your customer base and will want to set up survey alerts and email triggers so that you can respond directly to dissatisfied customers to address their problems; don't then tell them in the invitation that their survey responses will only be used when aggregated with other responses.

Similarly, do not offer incentives that you can't deliver. Again, this tends to happen by mistake, when an organization promises every respondent a reward, then has far more awards than imagined. See last week's post for strategies for survey incentives.