From DIY Surveys to DIT (Do It Together) Surveys

Panel management provides an excellent opportunity to improve the quality of the Do It Yourself survey research done by your coworkers. The initial problem of oversurveying customers has led to a resultant decline in response rates. Once your organization realizes it has a problem, a tragedy of the commons of shared respondents, a panel manager can begin overseeing that commons for sustainability, empaneling customers and controlling access to them for surveys.

The panel manager's primary role, relative to coworkers, is scheduling and allocating access to subsets of the panel for particular survey projects. This now provides an excuse for reviewing the survey before it is fielded and provides an opportunity to improve it. The panel manager can talk with the author to better understand the goals of the research and then review the questionnaire to make sure it can meet those goals. The panel manager can also review that the survey follows best practices for questionnaire design and that the survey invitations are well crafted. All of a sudden the panel manager is collaborating with the Do It Yourselfers helping them to do a better job.

Panel management can be the first step towards enterprise feedback management and a move from DIY surveys to Do It Together surveys.

Research News Roundup: Dreams, Teens, Social Media, Perils & Promise

News gathered from around the researchosphere this past week: Communispace and Ogilvy have researched the changing American dream, Carol Philips has blogged about the global teen, and Chadwick Martin Bailey and iResponse have studied the social media spleen (er, the effect of Facebook fan membership on WOM and buying patterns). Businessweek warns of the perils of market research, and Marketing Magazine says it's never held richer promise.

And now our top stories.

Communispace and Ogilvy have just published a new study of American consumers called 'Eyes Wide Open, Wallets Half Shut':

Among the study's key findings is that "having it all" is an unrealistic goal with 75% of those surveyed saying they would rather get out of the rat race than climb the corporate ladder - and instead, 76% said they would rather spend more time with family than make more money. Moreover, Americans are showing disenchantment with the pursuit of money with 75% again saying they would trade job security over a job that offered an opportunity for raises.

"The most surprising thing about our study was how much consumers were saying what they would NOT do for money, even when money worries are high on the list," explained Graceann Bennett, Managing Partner and Director of Strategic Planning at Ogilvy & Mather Chicago. "Prioritizing your life based on money is seen as a sure way to be disappointed..."

In fact, the recession has revealed important new consumer priorities with quality of life and peace of mind at the top and a focus on living life in a more sustainable way both from an environmental and financial point of view

Carol Phillips, in her blog post Where Teens Hang Out Digitally, sums up a wide range of research into the secret life of the global teenager. Teens multitask and forego sleep to cram in media consumption (20 hours a day when you double- and triple-count hours spent multitasking). Carol quantifies how teens text more than they talk and skip Twitter but use Facebook. What Outlook is to the businessperson's life, Facebook is to the teenager's life: a central site for messaging, calendar and contacts.

Speaking of Facebook, a Response magazine article, Facebook and Twitter Makes Consumers More Likely To Buy, Recommend, discussed recent research by Chadwick Martin Bailey and iModerate Research Technologies:

Consumers who are Facebook fans and Twitter followers of a brand are more likely to recommend and buy those brands then before they were fans and followers....60 percent of Facebook fans and 79 percent of Twitter followers are more likely to recommend those brands. It also revealed that 51 percent of Facebook fans and 67 percent of Twitter followers are more likely to buy the brands they follow.

"While social media is not the silver bullet that some pundits claim it to be, it is an extremely important and relatively low cost touch point that has a direct impact on sales and positive word of mouth," says Josh Mendelsohn, vice president at Chadwick Martin Bailey.

In other news, The Perils of Market Research, a Businessweek article by Steve McKee, offers these five pitfalls to be aware of:

1. There are things you can measure and things you can't. Don't mix them up.
2. Just because you can't measure it doesn't mean it's not real.
3. The best research is the real world.
4. People lie.
5. If you blindly follow the research, you'll lose. 

From perils to promise, where Marketing Magazine provides a comprehensive but succinct look at the "endless insight opportunities" that await MR in Tell Me About It:

Market research is attempting to change from an interrogative discipline into one that is more conversational. Instead of asking questions that prove what they already know, canny marketers and their research partners are simply opening their ears to listen to what people are saying.

'Two changes have come to a head at the same time,' says Ray Poynter, managing director of The Future Place. 'The first - a slow-burning one - is that marketers, NPD, innovators, and even sales have been realising that the results of too much market research are slow, expensive, and lacking in the sort of insight that would enable them to make a real difference.

'The second change is that the conversations between customers have shifted from being ephemeral to being indelibly inscribed on the web.'

And that's the way it is, March 20, 2010.

Innovation Jams

Most of the attention on MROCs has been for permanent standing communities, but Ray Poynter has written about short-term research communities and General Mills is moving to communities that last six to eight weeks. How short-lived can a community be?

Innovation jams grew out of an internal IBM technique for collaboration. IBM has established what you could call flash communities that last just three days (flash as in flash foods or flash mobs). For instance, in September, Lotus General Manager Bob Picciano invited customers, employees and thought leaders to the Smart Work Jam, "a global discussion on how to create a smarter, more productive work environment". Topics included how workplace teams will evolve, how to maximize the talents of the upcoming generation of employees and collaborative business process management.

While jams are short in duration, they are large in participation. The IBM Innovation Jam in 2006 had 150,000 participants drawn from employees and business partners in 104 countries. The jam generated 10 new business ideas that were funded with $100 million.

The jam process involves:

• Extensive marketing prior to the event to get people excited to participate.
• Facilitators and subject-matter experts to keep the discussion moving along.
• Researchers and analysts to provide real-time feedback on common trends and patterns in the discussions. are most successful when they leverage an existing real-world community, such as a community of employees and partners. Obviously, not much of a new sense of community can be built in three days among online participants.

Think of the innovation jam as the brainstorming part of a project plan. Instead of having a standing idea community constantly collecting thousands of new ideas, far more than you can implement, running a jam focuses everyone on a shared process at a point when it is most critical to the sponsoring organization.

Turn to a flash community like an innovation jam for that flash of inspiration.

Bullet Time for Matrix Questions

Among many other things, the movie The Matrix is known for its popularization of bullet time®, a special effect where time is slowed so that you can actually watch bullets move through space and see the characters dodge them. When a survey respondent tries to answer a matrix question (if they don't dodge them), time seems to slow as well, as they work through tedious row after tedious row; they actually answer the questions quickly, but it doesn't feel that way.

Matrix questions are compact and visually efficient, but they suffer from two primary problems.

1. A long record of research on research shows that matrix questions increase satisficing and survey abandonment (Krosnick, Narayan & Smith, 1996; Comley, 2000; Schonlau, Fricker, Elliott, 2002; etc.). A prominent type of satisficing behavior in matrices is known as straightlining, selecting the same column in row after row; this requires less mental and physical effort than choosing the most appropriate answer for each row. Many respondents will select a column, check the radio button in most rows in that column, varying to the left or right only for a few items. The result is a reduced range of differentiation between answers.
2. Earlier research that seemed to demonstrate that matrixes offer higher data quality (Couper, Traugott, Lamias, 2001) has largely been superseded by reports showing a loss of quality. Wide rows can visually confuse respondents about which item they are rating, as can the presence of too many columns (Gräf 2002). "We know respondents don't like grids," said Jackie Lorch, SSI VP, in a March 2009 press release. "They've been telling us that for years in focus groups and feedback, but we've always thought of grids as a necessary evil in questionnaire design. Now, we're beginning to learn that not only are grids frustrating for respondents - they actually produce inferior data." The SSI study found that a matrix refactored into many separate questions produced results with greater cohesiveness (.931 Cronbach's alpha compared to .799) and higher predictive validity (13.67 X2 compared to 3.75).

How bad are matrices? The blogger known by the alias ResearchRants collects bad examples in a category entitled "matrixes make me cry". Robert Moran of StrategyOne, in his Survey Research Resolutions for 2010, resolved to "Avoid the so-called ‘death grids' [large matrix questions] in online surveys." In an earlier post, he said, "No normal human has the patience to fill out a ‘deathgrid' and no non-incarcerated individual should be forced to do it."

In "Satisficing in surveys: Initial evidence", Krosnick, Narayan & Smith argue that you should re-factor matrix questions into simpler questions, an approach validated by the SSI study. Here are some possible refactoring strategies. Keep in mind that each of these will change the data quality and will introduce complications when comparing the new data collected to past results collected through matrices.

• Break each row of the matrix into a separate question or group of questions on its own page. Rewrite each row using the question types (e.g., choose one, choose many, open ended) as specified by the columns. In other words, the questions are the same but have been unpacked from the grid.
• Rewrite each row of the matrix into separate questions, replacing checkboxes with fully labeled scales. This was the approach favored by SSI, which produced high data quality but at the cost of taking twice as long to complete.
• Refactor importance matrixes into choose-many questions. This reverses the SSI approach, collapsing scales: For instance, rather than ask respondents to rate a list of attributes on an importance scale, ask them to select the most important items. While respondents are now only distinguishing between two classes of importance (important/checked and unimportant/unchecked) instead of five, in aggregate you will see wider differences between attributes than with the matrix question, and you will eliminate the possibility of straightlining. Respondents are now actually trading off which items are more important than others, an exercise they can avoid when filling in a matrix.
• Refactor Yes/No Matrix questions into choose-many questions (checkbox lists) instead. The respondent will not have to click a response for each item, resulting in less effort. Unfortunately, the two question types are not exactly equivalent: fewer boxes will be checked than radio buttons will be set to "Yes", according to Dillman, Smyth, Christian and Stern (2006).
• Be creative. Think about how you are going to analyze the data and see if other question types and skip patterns can produce similarly useful information with less effort from respondents.

Yes, its bullet time for matrix questions, as in put a bullet through as many of them as you can. 

Improving Response Rates to Web Surveys

My write-up yesterday of the Fan & Yan research about major factors that drive response rates omitted their advice about how to improve response rates, which I found to be more useful for academic research than commercial research. While Fan & Yan defined response rates as the percent of invitees who complete the survey, many researchers define response rates as the percent of invitees who start the survey. Because I consider start rates and completion rates to have very different drivers, in this post I am going to talk about getting respondents to at least start your B2B or B2C survey.

Fan & Yan did not describe the relative magnitude of how the different factors drive response rates. The following list is my judgment on the top seven factors, sorted by largest impact first. Most involve trading off some aspect of research quality in order to improve the response rate.

1. Target audience - Response rates are directly proportional to the strength of relationship between the survey sponsor and the target audience. Employees provide the highest response rate; lost customers and general market research the lowest response rates. This is not a lever you can easily pull to improve response rates, as the target audience is usually not something you can change. For some studies you may need to shift the target audience and redefine the research to improve response rates; for instance, if you had inadequate response from prospects, perhaps you have to settle for surveying customers instead.
2. Quantity of reminders - The most powerful tool you can use to improve response rates is simply to send out reminders. The first reminder creates the largest uptick in response; each subsequent reminder creates an increase but at a rate of diminishing returns. Too many reminders prompt invitees to unsubscribe from your email list and be unable to you for future research.
3. History of invitation frequency - This factor was not covered by Fan & Yan, but invitees who have been infrequently invited to surveys produce a higher response rate than those who have been frequently invited. Many organizations invite everyone from their house list of emails rather than using a randomly selected subset; if your house list is large enough, shift to inviting subsets instead of attempting a census. This is an important long-term step for improving response rates.
4. Empaneling - One group that can sustain a higher rate of survey invitations is a panel that you have specifically recruited for the purpose of periodic surveying. As with shifting your target audience, this can change your frame of research outside what you're studying, but is another pragmatic long-term approach.
5. Invitation salience - Survey topics that are highly of interest have greater response rates, and surveys of low interest have low response rates. The only survey I've ever conducted with a 0% response rate (for the first wave) had an invitation that specified the survey was about their most recent visit to the dentist. You want an invitation that best showcases the topic at hand, but don't misrepresent what the survey is about and don't be overly specific when that might bias the results. For example, sending out a survey invitation to measure the market opportunity for your new jetpack (Martin Aircraft to Introduce Jetpack) will overestimate demand in at least two ways: the topic is so cool that many will be interested in the survey, while those who are afraid to fly will probably skip it altogether.
6. Recency - To improve response rates on transactional surveys, move up the first invitation to within a day or two of the event being researched. The longer the lag between the event and the survey the lower the response rate.
7. Exclusivity - When you are surveying a random subset of your target audience, mention that fact in your invitation, emphasizing the exclusivity of the invitation, and you will see improved response rates. For instance: "You are one of a small, select group of customers that we have invited to provide us feedback." Of course, don't say it if it isn't true.

There are many other levers and switches you can pull to increase survey response rates, but these are the seven that seem to have the biggest influence on final results.

Response Rates Driven by 16 Major Factors

Weimiao Fan and Zheng Yan have published a comprehensive paper in the journal Computers in Human Behavior (March 2010) that synthesizes the results of 300 studies into a model of the contributing factors of response rates. While some researchers use response rate to refer to the number of people out of the target audience who start a survey, Fan and Yan use it, as others do, to refer to the number of people who complete the survey divided by the number of invited potential respondents.

The article "Factors affecting response rates of the web survey: A systematic review" provides a thorough summary of the many factors that can increase or decrease response rates.

Target audience

• Type of population: "General populations are found less willing to respond than employee populations, student populations, or army populations)." [See http://blog.vovici.com/blog/bid/18134/Survey-Response-Rate-Directly-Proportional-to-Strength-of-Relationship Survey Response Rate Directly Proportional to Strength of Relationship.] Within business populations, senior managers are less likely to respond than middle managers and staff.
• Demographics: Response rates vary by age and race, even after controlling for Internet access and computer literacy.
• Personality: Respondents are more likely to be conscientious, agreeable and open to new experiences than non-responders. Respondents "with a higher need for cognition" are more likely to start a web survey but less likely to complete it; respondents "with a higher emotional stability" are more likely to complete the survey.

Content of web questionnaires 

• Nature of the sponsors of the survey: Government and academic sponsors get higher response rates than businesses.
• What the topic is: Subjects that are of great interest to respondents have much higher response rates. Surveys on sensitive subjects have lower response rates.
• How long the survey takes to complete: "The length of a survey generally [has] a negative linear relation with response rates." 
• Question writing: Poorly worded questions and technical issues cause respondents to exit surveys.

Survey delivery

• Sampling methods: Unsurprisingly, panels provide higher response rates than email lists used for one-off invitations to surveys [because panelists have in effect agreed to participate in surveys].
• Contact delivery modes: Contacts include pre-notification of the upcoming survey, actual invitations and reminders. Email contacts reduce response rates when sent to abandoned email addresses or when blocked by spam tools. Research is inconclusive on whether increasing the number of delivery modes (for instance, sending SMS pre-notifications and sending non-responders a hard copy of the questionnaire) increases response rates.
• Contact quantity: "Several meta-analyses of both mail and web surveys have consistently concluded that the number of contacts is one of the most important factors to predict response rates." [The authors cite some circa 2001 research into the value of pre-notifications, but more recent research I've seen shows no effect for pre-notifications.]

Invitation designs

• Personalization: Some types of personalized email invitations improve response rates while others (e.g., personalized greetings, personalized email addresses) fail to. As a side effect, personalization does increase the risk of social desirability bias.
• Scarcity: Mentioning to recipients that they were part of a small selected group to be chosen to participate in this research increases response rates.
• Access control: Of three types of logins tested (username and password, only a password, or an automatic URL with a unique identifier embedded), the password-only login resulted in higher start rates, higher response rates and less socially biased answers.
• Use of incentives: Adding incentives did not significantly improve response rates, according to two studies (Goritz, 2006; Porter & Whitcomb, 2003b). [I think this varies significantly based on the type of survey and the relationship between the sponsor and the targeted respondents.]

Technology issues

• Survey software: Incompatibilities between survey software and different browser configurations [especially mobile browsers] can lead to reduced response rates. "Because of these variations, some respondents may not be able to...submit their answers successfully."
• Internet transmission capabilities: Intermittent Internet access can also lead to reduced response rates [again, especially a concern with mobile surveys].

This paper serves an important role in inventorying the many factors that affect response rates to web surveys. Tune into tomorrow for practical suggestions on how to improve the response rates to your own surveys.

Update: Improving Response [Start] Rates to Web Surveys (3/17).

Seeing the Elephant: Top-Down Customer Feedback Management

As I wrote last month in my post The Nancy-Bruce Project, one of my New Year's resolutions (besides committing to grueling half hour sessions with the Stairmaster) was to work through Bruce Temkin's advice for building a best-in-class Voice of the Customer program. And I as work through his advice, I will blog what I learn, a la the film Julie & Julia.

The first item on Bruce's list of 7 keys to Customer Experience in 2010 was "Drop the executive commitment façade". When our CEO promoted me to vice president of customer experience, he was signaling his own executive commitment to our customers.

The second item on Bruce's list follows naturally. "Acknowledge that you don't know your customers." Of course, we all feel we know our customers, but the departments we are in and the roles we serve shape our knowledge. It's much like the story of the blind men and the elephant: one man touches the trunk and concludes it is a snake, another touches the tail and concludes it is a rope, a third touches an ear and concludes the elephant is a hand fan. Too often, we haven't stepped back to see all of our customer's experience. We don't know our customer holistically. Having an executive in charge of the customer experience provides your organization with someone who is in charge of assembling the big picture. At Vovici, we had extensive surveys (of course), all deployed on a common feedback platform (of course), but no one had looked at the process from the top down.

Confession is good for the soul. Once you admit you could know your customers better, you're ready to change. As I've applied Bruce's advice as head of customer experience at Vovici, we've launched some new initiatives to improve the way we obtain and use feedback from our customers. We've chosen to take a three-step process:

1. Customer Lifecycle Feedback - First, we wanted to understand how a customer experiences our company. We started by documenting the points of customer interaction - from the sales process through the first year to the renewal process. Once we had our customer lifecycle map, we used that to clearly identify where we wanted to capture additional customer feedback. We also standardized question wording and scales that had grown out of sync over time. In all, we are currently fielding seven new or revamped transactional surveys and a customer relationship survey. 
2. Customer Insight Team - Once the surveys were launched, we formed a cross-functional team to review the data and make continuous improvements to our VOC program. Again, making sure everyone sees the elephant for an elephant. This team is using the data to prioritize cross-departmental customer experience initiatives and assure that departments are actively addressing reported customer issues. 
3. Acting on Results - We are actively improving customer experiences in two primary ways. Cross-departmental initiatives are identified and implemented by the Customer Insight Team. Changes within departments are led by the department directors. Many cross-departmental initiatives are underway, including improving system integrations so customer-facing teams have all the information they need when they are working with a customer. Another initiative is to implement a tool which will automatically broadcast survey data reports and dashboards to stakeholders, presenting them with timely data to get to know their customers while improving processes within their departments. 

It's easy to get impatient, but Temkin knows what he is talking about. We are already starting to see the impact of getting to know our customers better through a more programmatic approach. For example, early survey data indicated that our customers were not completely satisfied about how we communicated maintenance periods and new releases. We made changes to how we communicate with customers, creating standard processes for each customer segment. Overall satisfaction and loyalty rates are trending up.

If anyone has examples of collaborating with department managers to incorporate customer feedback into new procedures, please let me know. Vovici customers should always feel free to contact me with any questions or concerns.

And, if the elephant in the room for your business is that you don't truly know your customers, step back and design a feedback program that will provide you the big picture.

As for me, it's back to that Stairmaster...

Service Recovery & MR

Market researchers have not traditionally concerned themselves with using surveys to intervene on behalf of individual customers, because of the need for respondent anonymity and the interest in measuring rather than affecting customer satisfaction. But applied feedback techniques require designing processes that include service recovery after a dissatisfied respondent completes a survey. This prevents negative comments from getting buried in the report and facilitates direct action while there is an opportunity to recover.

Similar to customer satisfaction surveys, social media research is not anonymous - when your monitoring tools report back feedback, you can identify people (or at least their online personas) and either thank them for that feedback or try to get them assistance if they need it. Some organizations are expanding their customer service teams to monitor social media directly, so that they can respond as issues are tweeted and blogged. The market research department is then free to collect the feedback simply for aggregate analysis.

But with surveys the MR department needs to make sure that customer service is included. Here are three past posts that provide more detail:

• Survey Triggers for Feedback Management - Typically, a web survey of customers is set up with triggers or alerts that are fired off to staff each time a respondent gives a low rating to satisfaction or loyalty questions.
• Transforming Transactional Surveys - Adding triggers to a traditional questionnaire starts the transformation from a survey project to a survey process. Here are four issues to consider when you add survey alerts to an existing survey.
• Examples of Survey Alerts - Finally, here are six quick examples of using triggers to integrate surveys into process workflows.

Even when research organizations practice service recovery, rarely do they talk about it. One of our customers recently made it a point of pride when presenting the overall satisfaction results to the wider organization. They discussed the number of instances where customer service directly contacted the customer to address an issue and shared some of the follow-up comments from those customers.

Service recovery and market research will walk hand in hand for customer research from this point on.

Loyalty Triad: 3 Separate Components of Customer Loyalty

One of the central premises of Bob Hayes' book Beyond the Ultimate Question: A Systematic Approach to Improve Customer Loyalty is that customer loyalty is best thought of not as one item but as three separate components: advocacy loyalty, purchasing loyalty and retention loyalty. Bob writes:

How well we are able to predict business performance measures depends on the match between the business outcome and the component of loyalty. Retention loyalty, for example, is useful for predicting defection rates. Advocacy loyalty, on the other hand, is useful for predicting revenue. [Purchasing loyalty is useful for predicting average spending per user.]

Companies need to do their research to fully understand how different loyalty measures correspond to specific business outcomes. Single, simple metrics [as opposed to multi-item scales] are fraught with error and can lead to the mismanagement of customers and, ultimately, loss of revenue.

Bob summarizes the three components of customer loyalty with this table, showcasing three different ways to grow revenues:

Outcome	Behavior	Measure
Increase length of customer life	Decrease churn/defection rate	Retention Loyalty
Increase size of customer base	Increase number of referrals	Advocacy Loyalty
Increase number of purchases	Increase purchase behavior	Purchasing Loyalty

This elegantly makes the case that one question is insufficient to measure the complexity of customer loyalty. 

See my discussion of Bob's earlier book, Measuring Customer Satisfaction and Loyalty, for a description of the questions to include for an Advocacy Loyalty Index, Purchasing Loyalty Index and Retention Loyalty Index (the latter of which was called Defection Loyalty in the prior book).

Want to hear Bob Hayes in person? Make sure to attend the Vovici Vision 2010 conference, where Bob will be delivering one of the keynote addresses.

Research Trends Roundup

Mobile research, access panels, online qualitative, social media research. What's driving these markets? What do bloggers see when they look in their crystal ball?

• Mobile research (from Reg Baker's series of posts, especially Mobile Research Conference 2010 - Final Thoughts)
• Accelerators
• Ethnographic research
• Research at the "point of impulse"
• Decline of CATI
• Linking online and offline advertising campaigns
• Inhibitors
• Limited ease of use as an interviewing platform
• Penetration of Web-enabled mobile phones similar to U.S. Internet penetration 15 years ago
• Diversity of platforms to optimize for (from Blackberry, to iPhone, to Android, etc.)
• Obsession with the technology
• Online access panels (summary of Ray Poynter's post, What is the future for Online Access Panels?)
• Accelerators
• Shift from telephone and face-to-face surveys
• Integration with Do-It-Yourself survey tools
• Inhibitors
• Rise of proprietary panels and community panels
• Shift to mobile research and social media research
• Small loss to MROCs
• Market Research Online Communities (from Ray Poynter's post, How big is online Qual?)
• Ray estimates only 4% of current qualitative research globally is online
• Since only 14% of research is qual, that is just 0.6% of the overall MR market
• Online qual research is disproportionately U.S. (50-66% compared to U.S. research being 30% of the global pie)
• MROCs have surpassed "traditional" online qualitative techniques such as online focus groups
• Social media research (from Zachariah Hofer-Shall's post, What Is Social Intelligence?) - Applications:
• Rich qualitative insights from blogs and microblogs
• New quantitative metrics such as brand mentions and share of voice
• Assess success rate of interactive marketing campaigns
• Identify individuals needing customer service assistance
• Collect suggestions to improve products and services

In Perfect Tango: Social Media and Smart Phones, Harish Kotadia, Ph.D., discusses potential synergies that will fuel mobile research and social media research.

• Only 13.4% of the 650 million mobile subscribers worldwide have accessed a web page from the device in the past month. This usage will "skyrocket" in the coming years.
• When they do check the Web, the most popular type of site is social networking. Individuals will be providing feedback to their friends at the "point of impulse".
• Two thirds of mobile users are interested in receiving data tailored by their geographic location and social network.

Abundantly clear from all these discussions in the past two weeks: market research is undergoing as much change as it has anytime in the past 10 years. We'll need to keep the crystal ball handy.