Survey Test Mode

In a similar way, many users of our survey software dislike the fact that the first time they publish a new survey, it goes into test mode.  This is a not-so-subtle reminder that they should double-check the survey before inviting participants to respond.

Self Test

Pre-Test

That last question is particularly hard to answer in a self-test.  When I am feeling very unsure of a study, or the results are strategic rather than tactical, I will pre-test it on coworkers or, even better, on a small sample of the target audience (no more than 50).  I will end the survey with some questions about the survey itself, to identify areas or survey structures that were confusing or ambiguous. 

Pilot Test

Publish

Common Scales to Use when Writing Questions

One of the most frequent mistakes I see when reviewing questionnaires are poorly written scales. Novice survey authors often create their own scale rather than using the appropriate common scale. It's hard to write a good scale; instead you are better off rewording your question slightly so that you can use one of the following.

Acceptability	Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable
Agreement	Strongly disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Strongly agree
Amount of Use	Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness	Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate
Awareness	Not at all aware, Slightly aware, Somewhat aware, Moderately aware, Extremely aware
Beliefs	Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe, Very true of what I believe
Concern	Not at all concerned, Slightly concerned, Somewhat concerned, Moderately concerned, Extremely concerned
Familiarity	Not at all familiar, Slightly familiar, Somewhat familiar, Moderately familiar, Extremely familiar
Frequency	Never, Rarely, Sometimes, Often, Always
Importance	Not at all importance, Slightly important, Somewhat important, Very important, Extremely important
Influence	Not at all influential, Slightly influential, Somewhat influential, Very influential, Extremely influential
Likelihood	Not at all likely, Slightly likely, Somewhat likely, Moderately likely, Very likely
Priority	Not a priority, Low priority, Medium priority, High priority, Essential
Probability	Not probable, Somewhat improbable, Neutral, Somewhat probable, Very probable
Quality	Very poor, Poor, Fair, Good, Excellent
Reflect Me	Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me
Satisfaction (bipolar)	Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied
Satisfaction (unipolar)	Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Extremely satisfied

This list follows Krosnick's advice to use 5-point unipolar scales and 7-point bipolar scales.

Let me know any of your favorite scales that I omitted.

Hurdles in Race to Turn Recipients into Respondents

You've chosen the list of people you want to invite to your survey.  You still have a lot of hurdles to jump to get each recipient to actually take your survey:

• Make sure their email address is not on the unsubscribe list
• Keep your survey invitation from being flagged as spam
• Craft a survey subject line and From field that help get the invitation opened  
• Cover the six points important to respondents in an invitation
• Write an invitation that is compelling enough that recipient clicks on the link to take the survey

Representative Web Surveys Require Good Email Lists of Customers

Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.

Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don't, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.

Organizations with unrepresentative email lists need to take the following steps for web-survey success:

• Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
• Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
• Conduct a telephone study or paper survey of customers for which your organization doesn't have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
• Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication. 
• Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.

What steps have you taken to make sure that your web surveys are representative of your customer base?

Getting Your Survey Invitation Opened

Your survey invitation gets past the spam filters and shows up in someone's email inbox. Now the recipient will have two or three items at hand to decide whether to click Delete or to open your message:

• From field - The From field consists of the sending email address and the alias for that address. Nothing will get your email deleted faster than having an impersonal email address without an alias, such as - a particular favorite! - noreply@example.com. Always include the alias. What type of alias works best varies by organization; examples to experiment with for your invitations include:
  • The name of an individual - The name of an employee managing the survey or commissioning the survey can be very effective.
  • The name of the brand or company - While this can be impersonal, it may outperform the names of individuals for B2B vendors.
  • Brand name + "Surveys" - For some organizations a From field such as "Acme Surveys" works best.
  • Hybrid - Some panel companies test combinations such as "Acme Surveys & Jane Doe".
• Subject line - The text of the subject line will have the biggest impact: "Your Feedback Needed", "Some Quick Questions for You", "Help Us Serve You Better", "Learn What Your Peers Think About -----" are all subject lines that have been successful for survey invitations. Your organization will need to experiment to determine what works best for your audience.
• First line of the invitation - Some email programs display the first line of the invitation in grayed-out text immediately after a bold subject line. All the more reason to pay special attention to the copy you use in the body of your invitation.

What tactics have improved your organization's open rate on surveys?

Types of Questions: Four Building Blocks for Constructing Questionnaires

Sift out the finer distinctions and most questionnaires are constructed from four basic building blocks. These four, fundamental question types are:

Open-ended questions	Essay Questions	Long text responses
	Fill-in-the-Blank Questions	Short text responses
Closed-ended questions	Choose-One Questions	Single choice selected
	Choose-Many Questions	Multiple choices selected

Essay Question

The essay question, in a web survey, is displayed as a multi-line TEXTAREA: 

Respondents can enter a few words, a few paragraphs, and even a few pages. Typically essay questions store 32,000 to 64,000 characters of text. We've had respondents copy and paste dozens of pages of text into essay questions.

This question type is best used for understanding in detail what a respondent believes in their own words.

Fill-in-the-Blank Question

A fill-in-the-blank question is displayed as one or more text boxes with short labels, and is designed for gathering short responses:

It can also be used for contact information or address information:

Responses to text boxes are often validated to follow a common pattern.  Common validations include:

• Email address
• Whole number within a range
• Real number within a range
• Date
• Telephone number - U.S./Canadian format or international format
• Zip codes and postal codes

Choose-One Question

The most common question type is that of the single-select multiple-choice question, where a respondent chooses one and only one of the available options. Choose-one questions are typically shown with radio buttons or dropdown boxes.
While choose-one questions constrain the choices of the respondent, they are much quicker to answer and much easier to analyze.  

Choose-Many Questions

The multiple-select multiple-choice question allows the respondent to check all the choices that are applicable to the question.  Most survey software uses the standard checkboxes of HTML forms to show these questions:
These four questions can then be combined together in matrix questions.
For best practices for each type of question, see these posts:

• Essay-question best practices
• Choose-one question best practices
• Choose-many questions  

Compelling Survey Invitations

You've gotten the recipient to jump all the hurdles to get to the survey except for the last one - reading the invitation and clicking on the survey link.

You now have the challenge of writing a short but motivating invitation that will prompt your recipient to become a respondent.

As with any attempt at marketing, you have less than eight seconds to make a first impression. A clever use of images can help catch people's attention. You need to tell recipients what you want them to do as soon as possible in the invitation. Know your audience and write the invitation with this knowledge in mind. Keep the invitation short-but cover the key points.

Patrick Glaser's "A Playbook for Creating Survey Introductions for Online Panels" provides a useful outline for a survey invitation, along with example phrases.

I wish I could give you some proven catchphrases for subject lines that work with every survey, but what worked once--"Help us improve our products" or "Share your opinion"--might not work on the next round of survey invitations to the same audience. Experimentation and inventiveness are essential.

Do not make privacy claims that you may not keep. This tends to happen by mistake rather than from any intention to mislead recipients. Often you are surveying your customer base and will want to set up survey alerts and email triggers so that you can respond directly to dissatisfied customers to address their problems; don't then tell them in the invitation that their survey responses will only be used when aggregated with other responses.

Similarly, do not offer incentives that you can't deliver. Again, this tends to happen by mistake, when an organization promises every respondent a reward, then has far more awards than imagined. See last week's post for strategies for survey incentives.

Survey Software Success: Free EBook

The hypertext nature of this blog is great for when you want to dive in on one topic and follow links to related posts.  However, it's not well suited to those times when you want to read past posts in detail.

To address this, I've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The structure of the book was inspired by a presentation we did and outlines seven best practices for survey success:

1. Focus on a Goal
2. Survey the Right Number of People
3. Plan Your Mail Campaign with Care
4. Craft Your Questions Well
5. Order Questions Logically
6. Get the Survey Length Right
7. Close the Feedback Loop

Each of the above is its own chapter, and the book is about 73 pages long. You can use this form to get your free copy.

Ensuring Your Survey Invitation Isn’t Flagged as Spam

With the rise of Bayesian filters, most email users are seeing fewer spam messages in their Inboxes. You want to make sure that your survey invitation and reminder emails aren't flagged as spam, which would artificially lower your response rate.

Because spam filtering programs are constantly changing, it is best to send your draft email invitation to the personal email addresses of friends and coworkers simply to make sure all receive it. Be careful when discussing survey incentives in the email, as words and phrases such as "free" and "act now" are more likely to get your invitation classified as spam. Hectic punctuation - ALL CAPS, lots of exclamations!!!, dollar $ign$, L33t sp34k - are also more likely to get your message flagged.

Another important aspect of spam for U.S. organizations is the CAN-SPAM Act, which requires three types of compliance for organizations sending commercial email messages to organizations with which they have no relationship. (In other words, messages sent to customers do not fall under the jurisdiction of this act.) The three types:

1. Unsubscribe compliance - Each email contains an obvious and working method for the recipient to unsubscribe. All opt-out requests are processed within 10 days. The suppression lists of opt-outs are not mailed to.
2. Content compliance - The email From field is accurate, the subject field relates to the body of the email message and is not misleading, and the body contains a valid physical mailing address for the organization sending the message. (Surveys on sexual topics must be labeled in the subject header as containing sexually-explicit content.)
3. Sending-behavior compliance - Your email messages cannot be sent through an open relay and cannot contain a false header. Almost all hosted survey software applications comply with this. Additionally, your message cannot be sent to an email address that was harvested (downloaded and parsed) from web pages.

The penalty for failure to comply with these items ranges from a misdemeanor charge to an aggravated offense, depending on the size and scope of the offense. Where appropriate, you should consult a lawyer or review the CAN-SPAM Act in more detail to guarantee your email is in compliance.

What tools have you found helpful for writing email messages that will get past spam filters?

Reminder Invitations Double Survey Response Rate

We often argue for the need to respect your potential respondent's time and interrupt them as little as possible. Yet our survey software includes the ability to schedule the survey invitation and as many follow-up email reminders as you would like:

Reminders can be an annoying nuisance to recipients and should be avoided unless really necessary to your survey. Here are when they are necessary:

• You are surveying a small population (e.g., employees, major accounts) and have not yet reached your recommended sample size for 95% confidence.
• You are attempting a census instead of a survey. 
• You suspect that initial responders may vary in key ways from later responders. For some studies, when we run the numbers we find no measurable difference between early and late respondents; for others, we find important differences in satisfaction and loyalty.
• You are using email triggers to alert you to dissatisfied respondents and want to make sure that you have the opportunity to address their issues. 

Following up survey invitations with reminders is the most dramatic way to improve your response rate. For one recent study, a series of three reminders doubled the response rate:

Many of our users do not send out reminders because they are concerned about harassing potential respondents. However, since over 80% of responses to a closed-participation survey come within 24 hours of sending out an email invitation or reminder, think of that first reminder as doing the recipient a favor: often they meant to respond, but went on to other things, and the invitation got pushed down further in their in-box. Your reminder brings it back to the top of their mind - and their in-box.

One way to minimize the pain of reminders is to make sure that each recipient is given a unique hyperlink to the survey (this may be inappropriate in certain surveys where anonymity is vital). That way, reminders will only be sent to those recipients who have not yet completed the survey.

The only thing more annoying than receiving an email reminder about a survey you haven't taken is to receive a reminder to one that you have taken!