Survey Research Enterprise Feedback Management

Employee Engagement vs. Employee Satisfaction

Jeffrey Henning — Fri, 20 Nov 2009 14:35:00 GMT

Searches on "employee satisfaction" have dropped by half since 2004, while "employee engagement" has come from nowhere to surpass "employee satisfaction" searches in 2009.

Whenever I talk about employee engagement, I hear very different reactions:

"Employee engagement is just the latest buzzword for HR staff to embrace. It offers nothing materially different from employee-satisfaction research. It's just a way for consultants to sell last year's fashions at marked-up prices."
"Employee engagement represents an important breakthrough that links employee satisfaction to business outcomes. All employee-satisfaction research should become employee-engagement research."

One of the techniques I was taught for moderating focus groups is to assume that what the speaker says is true, from their perspective, and therefore to try to understand that perspective. So let's assume for the sake of argument that both of the above statements are true.

How might they both be true? Employee engagement makes explicit best practices for employee satisfaction that many researchers have followed implicitly.

Professional satisfaction research projects always focused on engagement. Employee-satisfaction research has traditionally been a measure of employees' emotional and rational satisfaction with their job and employer. Thorough researchers have always analyzed employee satisfaction as a key driver of employee retention, customer loyalty and other positive business outcomes.
Many satisfaction projects just measure satisfaction, rather than act on it. Far too many organizations do employee-satisfaction surveys simply because they feel they need to. They make tactical changes based on the feedback, but that's it. They aren't linking employee satisfaction to customer satisfaction; they aren't linking employee engagement to quality.

So, if you've done employee-satisfaction research strategically, employee engagement to you is just a new label for an old practice. But if you've done employee-satisfaction research tactically in the past, employee engagement represents a new best practice for you to aspire to in your research.

This evolution is analogous to how customer satisfaction research has evolved into customer loyalty research. Good customer satisfaction studies always looked at loyalty, but now that connection is explicit in the new label for the domain.

So for those of you with buzzword backlash, embrace the term employee engagement: point out the practices you've done in the past that differentiate such research from tactical employee-satisfaction studies, and help us all to do a better job conducting research that will help us satisfy and engage employees to create desired organizational outcomes.

SharePoint as Survey Software

Justin Corrado — Thu, 19 Nov 2009 16:21:00 GMT

We often have prospects looking to migrate to a survey software application from a home-grown system. Recently, we've been seeing more organizations looking to move from Microsoft SharePoint, an enterprise content management platform.

What clients liked about SharePoint when it comes to building online surveys:

The WYSIWYG form editor makes it easy to build basic surveys.
It offers complete flexibility for programmers to use HTML, JavaScript, AJAX techniques, Flash and other standard web development approaches.
It stores data directly in the repository that the administrator selects (e.g., SQL Server or Oracle).
Survey results are readily exported as CSV (Comma-Separated Value) or XLS files.

Here are the features that caused SharePoint users to move to a dedicated feedback platform:

It is difficult for business users to create surveys of moderate or significant complexity with SharePoint.
The lack of integrated email support meant that invitations, reminders and thank-you emails had to be created outside the system.
The lack of panel management meant administrators couldn't target respondents and track past survey activity.
It is difficult to personalize surveys based on known data.
SharePoint lacks survey-specific web services, making integration more tedious.
Some users reported security concerns.
Until recently, there was no support for basic skip patterns (available with MOSS 2007) or advanced branching.
It has no real-time reports or dashboards and can't export results to SAV or PPT files.

SharePoint is a great interim solution for organizations first adopting web surveys. It excels at short, uncomplicated and static surveys, but organizations outgrow it when they need advanced questionnaires, intermediate MR functionality, integrated email invitations, panel management or enterprise feedback management.

Panel Management Explained

Jeffrey Henning — Wed, 18 Nov 2009 09:55:00 GMT

A panel is a group of people with relevant backgrounds who agree to participate in surveys. Businesses can organize a panel for each group of key stakeholders: customers, employees, resellers, partners, prospects, etc.

Because panelists agree in advance to participate in surveys and feedback efforts, they become almost a guaranteed source of information for the sponsoring organization. Customers typically participate in the panel because they value their relationship with the sponsor, and they appreciate the additional information, influence and early access that comes from participating in the panel. For general market panels, panelists instead often look to merchandise and cash rewards, though even for many general panels people participate because they want to make their views heard.

How does a panel work? You invite an individual to participate in a panel and communicate the ground rules:

What they will be surveyed about
How you'll use the information
How frequently they'll be asked to participate in surveys
Why it's important that they participate
How they can opt-out if they change their mind
What's in it for them

Once you gain their permission, you invite the panelist to complete a registration survey, which will gather detailed demographic or firmographic information. This information can be used to target individual surveys to panel subsegments and also provides for rich opportunities in cross-tabulating survey results.

In contrast to online communities, in a panel, members communicate only with the sponsoring researcher, through the medium of the surveys they are sent. In online communities, members can engage in discussions with one another through an online portal.

Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, Customers as Confidants: Customer Panel Management Made Easy.

Vovici Survey Software and Web Survey: Request a Call l Request a Demo

Customer Experience Management Now a Core Differentiator

Brian Koma — Mon, 16 Nov 2009 13:14:00 GMT

Interest in Customer Experience rose dramatically across organizations we surveyed as part of our Vovici CE IQ study, increasing from 47% of respondents stating that Customer Experience Management (CEM) was very important to them in 2008 to 63% in 2009. Clearly the global economic contraction focused many organizations more tightly on understanding how customers are interacting with them, and this has put a dramatic new emphasis on CEM.

In fact, much to our surprise, more than half, or 55%, of respondents say that customer experience is a core differentiator that distinguishes their business from others in their market. Customer experience has arrived, and is now as important as traditional differentiators such as quality, service and staff capabilities. Despite the global economic contraction, price was not frequently cited as a core differentiator (18%).

When asked to assess the current benefits of CEM, customer satisfaction and loyalty were the most important, cited by 52% of respondents. Next was providing greater positive word of mouth (50%) and excellence in customer service (42%).

Leading the areas where organizations could do better, providing greater consistency of experience at every customer touch point was mentioned by only 36% of respondents. This is an area for improvement, as excellence in Customer Experience Management should drive such behavior. Top-line and bottom-line benefits from CEM are currently experienced by only about a third of respondents (35% and 29% respectively).

With organizations acknowledging how important CEM is to distinguish themselves from competitors, it is more important than ever to focus on those CEM best practices that build customer loyalty.

Note: The survey of over 200 organizations around the world is based on a convenience sample designed to identify the impact of CE best practices on loyalty, in order to prioritize those best practices. These results are illustrative and are not representative of any wider population of organizations. For more on the methodology, see Customer Experience Study Findings.

Request a Call Vovici Survey Software and Web Survey Request a Demo

Customer Loyalty Questions

Jeffrey Henning — Fri, 13 Nov 2009 09:11:00 GMT

When asked to measure loyalty, what questions should you ask?

For instance, in The Value Profit Chain, Earl Sasser Jr. and his co-authors provide this list:

How likely are you to repurchase?
How likely are you to tell someone else about your experience?
How many times have you purchased (all products or services of this type) in the past 6 months (or some other period of time reflecting purchase frequency for product or service category)?
How many times have you purchased (this product or service) from (our company) in the past 6 months?
How many others have you told about your experience (with this product or service) in the past 6 months?
How many of those that you have told in the past 6 months have, to your knowledge, also purchased (the product or service)?
How many times have you offered constructive criticism or suggestions for product or service improvements over the past 6 months?

To this list can be added:

How likely are you to repurchase if the price increases 10%?
How likely are you to repurchase if a like competitor has a price that is 10% lower?
How vital is the competitive advantage this product provides?
What percentage of your spending on this product category is spent with us?
When shopping for this product category, how often do you purchase from us instead of from other brands?
How likely are you to continue purchasing the same products/services from us?
How likely are you to purchase different products/services from us?
How likely are you to increase the frequency of purchasing from us?
How likely are you to switch to a different provider?
How reluctant are you to switch your business to a competitor?

You should measure the questions that you can link to business outcomes. This takes time and experimentation to find out what works best for your organization. Settling on one question prematurely because it works for others in a few industries is the wrong approach. For you, one set of questions might work better for one product line than another, which might need a completely different set of loyalty questions.

Here are some sets of loyalty questions you should test:

Survey Compensation for Employees Gone Awry

Jeffrey Henning — Thu, 12 Nov 2009 12:28:00 GMT

I'm adamantly against financial compensation for staff based on customer-service survey results. Invariably, it leads to customer-service agents gaming the system; this happens far more than executives are willing to acknowledge. For instance, a K-mart cashier posted a sign at the register saying, "For a chance to win a $2,500 gift card, log onto KmartFeedback.com and rate our customer service 9 or 10. Thanks for shopping at your 1 Penn Plaza Kmart!" Of course, people have a chance to win no matter what rating they give.

(photo: bcurrie)

A post on this example at The Consumerist led to a rash of comments giving examples for other major brands:

CVS - "Yesterday I went to CVS for a prescription. When I checked out, the cashier looked at my receipt and stated that I had been selected to participate in a survey. Yeah, yeah, yeah, I know - this happens all the time, but she then showed me a mini candy bar and said, ‘you get a free Take5 candy bar so that will remind you to give us all 5's on the survey!' I thought it was pretty smart but I still didn't do the survey..."
Enterprise - "I was asked to rate an Enterprise rental by the customer rep, on a scale of 1-10 (or whatever it was). I said 9 because I figure one should save 10 for over-the-top service; this particular rental went smoothly, but was nothing special. He kept after me, wanting to know what was wrong, why I didn't give them a 10, until I got the point that to them ‘10' means ‘normal service'."
Home Depot - "The Home Depots around me [hand you a flyer] telling you the almost exact same thing."
Nissan - "This reminds me of whenever I buy a car. The Nissan salesperson reminds me that I will receive a survey in the mail, and that I'm supposed to return that survey with all 5's (the best possible mark), or else their dealership will be dinged or knocked down some points or whatever. I don't generally return surveys anyway, and I certainly don't when someone tells me what my answers should be anyway."
Old Navy - "An Old Navy store stapled a piece of paper to my receipt with a similar suggestion - something like ‘rate us a 10 and get 10% off'. Of course you get 10% off for filling out the survey no matter what, so I filled out the survey honestly, with perhaps a slight negative influence from the suggestion."
Sears - "Did the same thing when I worked at Sears but it said all 10s and we were told to tell them all 10s or we'd fail. It would be nice if they used the surveys to better themselves."
Target - "I have had plenty of cashiers at Target say that sort of thing too."
Toyota - "A Toyota dealer actually told me to bring in the survey so we could 'fill it out together'. In exchange - he would fill my gas tank."

My recommendations:

Don't financially incentivize any staff based on survey results. Doing so will change the results, no matter how strong your corporate culture is (for instance, Enterprise would discipline the representative mentioned above). Compensating only managers will lead to managers asking representatives to ask for higher scores. The most important thing is to gather authentic feedback about current satisfaction.
Don't use survey results as a crutch for measuring employees. It's vital that you share rich, relevant feedback with agents so that they can serve customers better in the future. Their managers need to mentor them not overmeasure them. (See Employee-Customer Engagement Best Practices.)
Don't use the Net Promoter Score as a transactional measure. From these and other anecdotes, it is clear that the NPS in customer-service settings leads to a collapse in the range and validity of the scale and an obsession with the top score, even for organizations like Enterprise that try to counterbalance this.
Don't use numbers. They are arbitrary; in some pure mathematical realm they are better for averaging than the results of fully labeled scales but the lower interrater reliability of numeric scales negates that. What an 8 means on a 10-point scale varies from respondent to respondent. Use labels, as in these common rating scales, and make the highest labeled point hard to achieve: Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied.

Do you have any examples of being asked to provide artificially high ratings to surveys?

Vovici Forms Foundation of Oracle Voice of the Customer Program

Jeffrey Henning — Wed, 11 Nov 2009 12:42:00 GMT

Jeremy Whyte, director of customer feedback and reporting with Oracle Corporation, presented details of Oracle's extensive Voice of the Customer research program to the American Marketing Association in a research webinar on October 20. Surveys form the foundation of Oracle's Voice of the Customer program: listening to customer input through Vovici surveys provides "comprehensive feedback across the Oracle ecosystem and customer ownership lifecycle". Oracle conducts hundreds of surveys, grouped into relationship surveys, transactional surveys and targeted surveys.

Relationship surveys with customers, partners and employees are the most strategic surveys, highlighting customer experience and loyalty drivers across cumulative contacts. Why survey employees in a Voice of the Customer program? Because Oracle has identified a correlation between employee satisfaction and customer satisfaction and prioritizes investments to improve employee satisfaction based on how that will drive increased customer loyalty.
Transactional surveys measure the quality of each service response by organization. Short term these surveys trigger immediate action on a customer-by-customer basis, and long term these drive operational improvements to improve service quality. Transactional surveys are conducted for technical support, customer service, consulting services, education services, sales win/losses, implementations and even M&A impact on customers.
Targeted surveys are used primarily for qualitative research to supplement relationship and transactional surveys. Sample applications include competitive intelligence, product and service planning, marketing, referencing, user group satisfaction and general market research.

All of these survey results are combined with operational measures and financial outcomes in a customer satisfaction linkage analysis. As a foundational platform, taken together these surveys empower Oracle's wider Voice of the Customer program, which reaches beyond surveys for other additional types of feedback.

How effective has this been? Oracle has improved customer satisfaction each fiscal year for five years running and its customers now demonstrate the greatest propensity to recommend that they ever have. If you talk to a large Oracle client, you will hear firsthand how they have seen Oracle adapt and improve to serve them better. To achieve similar results for your own organization, make surveys the foundation of your efforts.

Insights from the Gartner CRM Summit

Jeffrey Henning — Tue, 10 Nov 2009 15:21:00 GMT

This year's North American Gartner CRM Summit highlighted important changes in technology and innovation, many driven by the rise of social media and social networks.

If you divide the CRM market into Operational CRM, Analytical CRM and Social CRM, as Gartner does, then Social CRM has the smallest share of current spending (10%) but the highest share of growth in spending (60%). In fact, Gartner analyst Michael Maoz said that "Networks matter more to your success than any other initiative."

When Jim Davies gave his "State of EFM" address, he too emphasized the rise in social technologies, emphasizing the integration of social feedback within enterprise feedback management, noting that market-research online communities are now a part of many EFM platforms, including Vovici. For Jim, the leaders of the EFM market remain Vovici and Confirmit, and he expects continued consolidation in the industry (Vovici has purchased three companies: Perseus, WebSurveyor and Surveyo, as well as Raosoft web-survey assets).

Not all social technologies encourage innovation, though: research director Gareth Herschel reported that Gartner clients had seen marginal-to-no ROI from investments in blog content analysis, and Gartner instead advocates that its customers learn the language of the Voice of the Customer by asking more open-ended questions in surveys and then text mining the responses. As you innovate, you need to monitor your progress, and Herschel emphasized the need to select 5-9 key performance indicators from the hundreds of performance indicators your organization follows. In a separate presentation, Herschel said that organizations need to engage in meta-analysis, analyzing their analysis process, to make sure that they improve the quality of decisions that they make in the future.

Kathy Harris, a Gartner distinguished analyst, provided a good summary of the conference. She asserted that "CRM is the most fruitful innovation opportunity for your company and your customers"; make sure to use EFM (which Gartner sees as a subset of CRM) as you "embrace your customers in co-creation."

Social Media Market Research: A Study of the Tropicana Repackaging

Jeffrey Henning — Mon, 09 Nov 2009 11:29:00 GMT

At the 2009 MRA First Outlook Conference, Owen Shapiro of LJS Associates and Tom Malkin of GeeYee, Inc. presented "The Impact of Social Media on Market Research". In order to better understand the strengths and weaknesses of social media research, LJS conducted phone and social media research into the Tropicana unrepackaging incident.

Earlier this year, Tropicana rolled out repackaging that many shoppers complained looked like a generic brand and made the product much harder to find on store shelves. The resulting pushback led to Tropicana to revert to its earlier packaging.

To study this, LJS surveyed 1,000 U.S. consumers by phone and online and then weighted the results to be demographically representative. Fully 20% of consumers surveyed had noticed the new Tropicana packaging; of these, 32% liked the packaging, 32% were neutral and 27% were negative. Only 1% of consumers had posted online about the packaging.

For social media research, GeeYee scraped 1,900 posts from over one million web pages analyzed; each post typically covered three or four topics. Here is the categorization of the most frequently mentioned topics:

Price - Primarily discussions of coupons and sales
Premium OJ - Discussions of comparatively higher cost
Ingredients - For instance, the shift to Brazilian rather than Florida oranges
Confusion in store - Fully 34% of posters discussed not being able to find the product in stores
New package design - Discussions of this accelerated once the packaging was withdrawn
Reversion to "classic" packaging - Posters commented about Tropicana's announcement or upon seeing the old packaging restored

Comparing surveys to social media is like comparing apples to orange juice (sorry):

Social media's impact is disproportionate to the level of activity. The multiplier effect for each posting produces an impact often missed by traditional marketing metrics. Lost in a typical survey is how vocal those who disagree might end up becoming.
The data is dirty. When it comes to social media, you can't distinguish between grassroots and "Astroturf" (sponsored marketing campaigns design to mimic grassroots efforts), between legitimate blogs and "splogs" (spam blogs), between memes being relayed because they resonate or because they reciprocate (blog-post exchanges).
Brands with unique names are easier to research. It is easier to extract quotes about the iPhone than Coach. Researching Tropicana required removing references to the Tropicana casinos in Las Vegas and Atlantic City, a level of cleaning not needed in coding verbatim survey responses.
Topics are not always organized to answer our questions. Since posters can discuss anything, you need to develop a much broader coding sheet than for open-ended survey questions, which are tightly focused. Sometimes social media may not be talking about your question at all.
Social media is not representative, only directional. You can try to quantify it to better understand the direction but it will still be qualitative.

Owen concluded that social media is best used early in the market research process: to develop hypotheses to test with surveys or to use up front as a qualitative tool to more fully understand issues and to see what you might have overlooked.

General Mills Moving Qualitative Research Online

Jeffrey Henning — Fri, 06 Nov 2009 14:10:00 GMT

Ned Winsborough, manager of consumer networks at General Mills, presented "Accelerating Innovation with Social Networks" at the MRA First Outlook Conference. "We have a mandate at General Mills to move as much of our qualitative research online as possible in the coming months and years. We have been experimenting with this for a year, but we created our consumer networks team this summer and are now scaling it." (Consumer networks is the term that General Mills uses for MROCs.)

General Mills has done 22 community projects since last spring. Why online communities? "Online consumer communities meet the needs of consumers, brand teams and agencies with busy lives. They allow you to innovate with consumers better, faster, and cheaper." With communities, General Mills is able to engage in iterative building of concepts: "We listen, we build; we listen, we tweak. This can be done very quickly, with a lot of flexibility to the method." Community research allows for faster speed to market. For one project, General Mills did six months of work in six weeks. Compared to other qualitative methods, communities are less expensive. "There is a fixed cost for setting up the communities, which can be very significant, but the incremental cost of doing extra weeks, extra moderation, is very low."

As a result of General Mills' 22 projects, they have made changes to their approach to community research:

Focus on Discovery - The General Mills innovation model uses three steps: Discover, Build, Launch. The communities are great for Discovery but less suited for the Build phase. In the Discovery phase, community research always works, according to Ned, whether the project is big or small, whether the tolerance for risk is high or low. In the Build phase, small projects can be supported with community research but larger projects require traditional quantitative research. For future community research, "we are focusing on Discovery."
Smaller Communities - Early communities were larger (for example, 225 participants), but that produced too much information to quickly and easily analyze. "Now we work with communities of 30 to 50 people (more if we have subgroups we want to investigate). With fewer members, we really get to know them as individuals, and we can probe better."
Shorter Duration Communities - General Mills has moved from a permanent online community to project-based communities that last for six to eight weeks. "This is a different model than creating one ongoing community. We have some experience with that type of community: we had done that in the past but found it wasn't cost effective." The ongoing moderation activities can be significant, yet "it is rare that we have things that we need to do every week."
Larger Incentives - Members to an early community were offered $50 for six weeks participation and a chance to win some modest prizes. Current incentives tend to run $40-50 per week.
Geographically Centered - For one of its first project communities, General Mills invited seven local participants to come to their facility for shelf tests and project packaging tests. Now, General Mills "uses focus group facilities to recruit members, so that we can do selective face to face research."

Ned has heard everything from "traditional research is dead" to skepticism about the value of online community research. "The truth is in the middle," he said. "It has a place, and we need to approach it like any other new technology. What questions can it answer? What objectives can it meet? What objectives can't it meet? Where can it fit in an array of methods? It certainly doesn't obsolete core quantitative methods but it has powerful potential to transform qualitative research as we know it."

MROC Case Study from ABC Studios

Jeffrey Henning — Thu, 05 Nov 2009 21:57:00 GMT

Karen Manne, VP of research with Disney, presented "Journey inside the ABC Studios Advisory Panel" at the MRA First Outlook Conference. ABC Studios is the production company for the Disney television group, producing shows such as Brothers & Sisters, Desperate Housewives and Grey's Anatomy, as well as Castle, FlashForward, Legend of the Seeker and Lost, among others. "We started building this community three years ago," said Karen. "ASAP (the ABC Studios Advisory Panel) was the first online community at Disney and the first for program planning in the TV industry."

The community currently has 1,900 members, who are each heavy viewers of at least two ABC Studios shows and who are opinion leaders: people who are passionate about television and regularly talk to their families and friends about the shows they watch. Membership fluctuates, as members who don't log in for at least three months are purged occasionally; a purge six months ago reduced membership to 1,400. Unfortunately, the panel is not gender balanced: 86% of the members are female; Karen recently went to ComicCon to unsuccessfully recruit more men for the panel, but most men only watch one of the ABC Studios shows.

Members are not given monetary incentives at all, but participate because they want to have a hand in shaping TV programming. They are sometimes given digital access to TV shows and also see sneak peeks of shows.

For a time, members could refer friends to the community, but Karen has stopped that practice. Too many referrals were skewing some of the research, as members invited others with identical views on particular characters or aspects of the shows.

Here are some anecdotes about the community by show:

Brothers & Sisters - The producers were interested in viewer opinions of several of the male characters; rather than tip their hand into which characters they were most interested in, they did a general study of all the male characters on the show.
Castle - As an entertainment company, ABC Studios is able to provide unusual rewards to its members. Three heavy contributors to the site were invited to a book signing of Heat Wave, a real book marketed as if it were written by the fictional Richard Castle (played by Nathan Fillion, of Firefly). These community members were given the VIP treatment and were photographed meeting Fillion; this was then publicized in the community.
Grey's Anatomy - One problem Karen has experienced is the occasional leak of sensitive information from the panel. A poll about the character Lexie was released to the public, for instance. As a result, for some sensitive polls, respondents are no longer shown the results. ("Polls are a favorite of members, since they're a quick and easy way to provide feedback.")
Scrubs - ABC bought the rights to this NBC show and is relaunching it with an altered premise this season. The season premiere was uploaded for community members, who were encouraged to watch the whole episode and provide feedback on the significant changes to the story. Initial viewer reaction was positive.
Ugly Betty - After three years of working at the fashion magazine, Betty is finally getting a makeover, and ASAP members reviewed seven possible new looks. Through leaks from the community, this lead to fevered coverage in blogs (Perez Hilton: De Uglifying Betty) and the entertainment press, and finally, even, the Wall Street Journal ("Making Ugly Betty Prettier: To gauge viewer reaction, ABC turns to online focus groups to test its star").

For ABC Studios, the benefits of ASAP are many:

Provides easy access to consumers
Yields quick feedback on insights and attitudes
"Spontaneity allows for flexibility"
Targeted research
Viral marketing

Many research projects are quite small and targeted, leading to shorter, more focused questionnaires. Karen has done literally a 1000 projects in the community.

Karen said, "I love my community but it is not all puppies and rainbows - it takes a lot of work." Some of the cons:

Unable to verify that members aren't reporters or competitors
Requires ongoing investment of time and money to recruit new members
Leaks of sensitive material to the entertainment blogs and press
Piracy of episodes posted within the community
Qualitative data is voluminous and time consuming to analyze
Busy work - during off-production times (such as the summer) need to have community activities to keep members engaged for when they are really needed
Victim of success - get pushed for rapid turnaround because executives realize the community enables it

For all its cons, Karen said the benefits outweigh the challenges. "Online communities are the hot new ticket in market research."

From Bedrock to MROC: Member Activities beyond Discussions

Jeffrey Henning — Thu, 05 Nov 2009 18:58:00 GMT

Jane Mount, PRC, an executive vice president with Digital Research, presented "From Bedrock to MROC: An Evolution in Qualitative Research Practices" at the 2009 MRA First Outlook Conference. Jane began by describing how quantitative methods over the past sixty years have evolved from door-to-door techniques, to direct mail, to telephone and to online research, while qualitative research has remained with focus groups during this time period. Even today online focus groups represent less than 10% of the qualitative market. MROCs, however, represent a substantial shift in qualitative research: "a shift from asking questions to get reactive consumer feedback, to listening to dialogue to get proactive consumer insight."

Jane provided an excellent introduction to the topic of MROCs, covering familiar ground. (If you're new to MROCs, see my past posts on MROC = Market Research Online Community, Focus Groups vs. Online Communities and Social Networks vs. Online Communities vs. Panels.)

Jane presented the MROC market as a continuum ranging from full-service suppliers such as her firm, Digital Research, on one side to technology-only suppliers such as Vovici on the other side. [She showed a couple of her competitors, and a couple Vovici competitors, all of whom I have happily omitted from my recreation of her slide!]

Frequently researchers think of MROCs as simply "listening posts", a place to facilitate discussions and eavesdrop on conversations, but - as a full-service supplier - Jane talked about the many other research activities that MROC members can do:

Static ethnography - Upload photos from their personal life.
Representational images - Upload clip art or a photo that represents a topic. For a sensitive topic like body image, have them submit these privately. For a fun topic like perceptions of their in-laws, have them upload the image for all to see and comment on (one member uploaded a picture of a cactus with the caption "They're prickly" to describe their in-laws, inspiring a comment thread from others who agreed).
Idea banks - Submit ideas to a shared database where they can rate them.
Insight games - Play word-association exercises and MadLibs-style sentence completion games.
Cartoon captions - Write a caption for a cartoon.
Personal diaries - Record daily activities, providing a richer narrative than possible through a one-time survey; for instance, revealing how members struggle with dieting on a day-to-day basis.
Collages - Assemble collages that represent the topic being researched.
Fun polls & quizzes - Answer entertaining questions like "If the economy was a candy bar, which of these candy bars would it be?"
Team activities - Do planned exercises with others. The research team segments users upon registration and then plans team activities where each team represents a different segmentation. For instance, DRI did a traditional quantitative study that produced six segments of consumer buying behavior, then invited those respondents into the community and recorded their segmentation.

Jane said that issues suitable for research with MROCs include "ideation; testing social media strategy; trend spotting; early stage evaluation of branding, packaging, ads; exploring attitudes and behaviors; directional insights when time is critical; and testing suitable language for a target." To her mind, MROCs are a very cost-effective method for qualitative research that is gaining in popularity because they are fast, provide ongoing insight generation, are highly creative, and their tech-intensiveness matches with respondent lifestyles today.

MROCs are now part of "a modern Stone Age family" of qualitative tools.

Multicultural Market Research: Eight Make-or-Break Rules

Jeffrey Henning — Thu, 05 Nov 2009 12:43:00 GMT

David R. Morse, president of New American Dimensions, discussed his new book Multicultural Intelligence: Eight Make-or-Break Rules for Marketing to Race, Ethnicity, and Sexual Orientation at the MRA First Outlook Conference. He argued that all U.S. researchers need to develop competency in multicultural research, given shifts such as these in American demographics:

The percentage of foreign-born population in the United States has increased to 14%, a level not seen since 1910.
Hispanic immigration is at an all-time high, and Hispanics will make up 20% of Americans by 2050.
By 2044, white Americans will be a minority according to projections from the U.S. Census department.

Here are David's eight rules:

Boost your multicultural competency. The U.S. Census is very detailed and useful for developing a detailed demographic understanding (though it lacks demographics by sexual orientation). When doing focus groups or qualitative research, make sure to have a moderator or interviewer of the race or sexual orientation being studied.
Divide and conquer. Don't accept the stereotypes ("Hispanics are brand loyal") but segment this population to truly understand how it relates to your market. The Current Population Survey is one of the few resources that lets you segment population by generation. Foreign born (or first-generation Americans) are very different from second-generation Americans, who generally speak English as well as their parents' native language; second-generation Americans are different again from the third-generation Americans, who typically speak only English.
Don't trust the experts. The accepted wisdom is often full of persistent and incorrect "truths" and urban legends, especially relating to translation. Do your homework.
Don't let the joke be on you. Tread carefully when doing multicultural marketing with humor; what is funny differs dramatically by culture.
Don't get lost in the translation. Given the prevalence of translation errors, make certain to backtranslate the questionnaire. David once was surprised to see in a survey that 100% of Spanish-speaking Hispanics disagreed that "the Internet is color blind". Rechecking the Spanish translation, he found that it has been translated to mean "the Internet has red-blue-green color blindness".
Push their buttons. Find the cultural cues that people resonate with; use your research to determine what those are for your market.
Market on a wink and a prayer. In a mainstream ad, insert a subtle cultural reference. The mainstream will miss it, but the targeted culture will appreciate it.
Make up; don't cover up. Watch the watchdogs. Many segments have advocacy and anti-discrimination groups; if your marketing runs afoul of them, stop the campaign at once and apologize.

America is not one culture, and savvy researchers realize this. "We are not a color-blind society," said David. "We live in different worlds. We talk differently. We listen to different kinds of music. We worship differently." Follow David's eight rules to improve your organization's marketing and market research.

Market Research at Microsoft: Evolution of the MR Department

Jeffrey Henning — Wed, 04 Nov 2009 20:35:00 GMT

At the MRA First Outlook Conference in San Diego, Reed Cundiff, senior director of central market research for Microsoft (and formerly an analyst with the Yankee Group), discussed how market research at the company has evolved and is evolving.

Prior to the creation of the Central Market Research Insights team, researchers had existed alone or in pairs in many different departments across the company. Job descriptions varied significantly; researchers had no career path within Microsoft.
Four years ago, the CMRI had just eight staff. It peaked at about 102 people in June, before being brought down by layoff to 97 staff. The upside of the downside, as it were, is that it has further accelerated the centralization of market research: departments that in the past were funding their own research are now turning to the central group instead. "We see that a lot of ad hoc research budgets have been cut; that is good for us, as there were many projects done outside of our research group." As a result, the organization is eliminating redundant and superfluous research expenditures.

The vision for the central research group is "to be a driving force behind Microsoft's business and product strategy" with the mission of delivering "strategic, fact-based insights that drive Microsoft's most essential business decisions."

The organization spends $80M to $110M annually on external research: the technology sector's largest research budget, according to Reed. Projects range from doing a market opportunity analysis for a v1 health care product to conducting a customer satisfaction survey with 100,000 respondents across 86 countries. CMRI devotes 3.5 FTEs to its research vendor management program, where they develop the preferred vendor list and do biannual reviews of the vendors (and ask the vendors to review the research managers they interact with). The result has been better use of outside vendors and consistent improvement. Before the new process, Microsoft would "run a pilot with a vendor, fall in love with that vendor, bury that vendor, then never do business with that vendor again!"

CMRI's strategy has three key components: to "deliver integrated insights" (primary research with market analysis), to "be a trusted advisor" and to "display business acumen". As a result, what Microsoft wants and expects from its internal researchers is changing [worth a blog post of its own!]: researchers need to be more consultative and need to specialize in a few focus areas.

In its research on research, the CMRI has adopted Six Sigma. "In the past fiscal year, we went through the Six Sigma process and we are reducing the number of defects study by study by study. We averaged 12 defects per final report in a six-month period to 3 the next period down to 2 most recently."

"We need to seize the opportunity. We are moving through a lot of challenges but to drive a fact-based culture, the timing has never been better for us." In his concluding remarks, Reed said, "We spend millions of dollars that affects hundreds of millions of dollars in marketing spend that affects billions of dollars in revenue. We have to get it right."

Market Research Regulation in 2010

Jeffrey Henning — Wed, 04 Nov 2009 17:38:00 GMT

At the MRA First Outlook Conference, Howard Feinberg (@hfienberg), director of government affairs of the Market Research Association, presented "Regulation Over the Horizon: Emerging Research Technologies & Modes and the Legal & Privacy Hurdles". Howard covered six broad trends that have implications for the future of market-research regulation.

Online behavioral tracking - FTC proposes self-regulatory standards, which means "you need to regulate yourselves, or we will do it for you". Facebook recently settled a lawsuit about Beacon, which was considered to be in violation of established, offline privacy laws. Public opinion about behavioral tracking is mixed: users prefer free web content, paid for by ads, to subscription websites and are aware that information about their web browsing is collected, but are disturbed by the "creepiness" factor when they are unsure about what information is collected or how it is disseminated. In a recent survey, 92% of respondents felt that there should be a law requiring websites and ad servers to delete all information about them upon request.
Location and behavioral data - A little appreciated downside of our connectivity is that we've given up locational privacy, as we are tracked by our cellphones and GPS devices in our cars (if you know someone's home and work address you can identify their GPS data stream); sites like Google Latitude and Loopt provide information about where opt-in users are in real time. Locational data is not just about where you go, since those locations demonstrate associations - "political, religious, amicable and amorous, to name only a few," according to a New York State Court of Appeals ruling in May. Even with GPS off, phones can triangulate location from cell towers. Other developments that provide the MR industry great opportunities for observational research from data aggregation:

Car insurance companies in California were recently given approval to charge rates based on travel patterns and mileage.
Feinberg joked about RFID tagging of research participants, pointing out that RFID-tagged passports are now used at U.S. border crossings.
A smart grid for power management of networked appliances and HVAC systems could provide insight into intimate consumer usage of appliances.
UK billboards photograph license-plate numbers to look up the make and model of the vehicle, showing an ad for the exact motor oil required by that vehicle.

Social media - Scraping social media is limited by concerns about data quality, which prompts questions about who these people really are. From an ethical standpoint, are users aware that you are watching, listening, reading, analyzing their output? Teens and tweens are often naïve about the privacy of their data. New laws are regulating social networks, without defining social networks; this could have implications for researchers, if online focus groups and online research communities become classified and regulated as social networks. [See Social Networks vs. Online Communities vs. Panels for my definition.]
Respondent authentication -One survey found that web users were open to authentication by trusted vendors. No U.S. law currently governs digital fingerprinting, which is used by many panels for respondent authentication. In Europe, an IP address may constitute protected PII (Personally Identifiable Information).
Cloud computing - The provision of data through servers accessible over the Internet raises risks for data security; Howard advised researchers to make certain to download and backup data stored on such systems. How do international data transfer laws relate to cloud computing? This is yet to be decided.
Ethical and legal initiatives - For self-regulation of the industry, researchers are urged to provide transparency and consumer control over data, with limits on data retention. A standard practice, widely adopted, is get consumer consent whenever a privacy policy is changed. An additional practice is to get consent for using "sensitive" data. Best practices for location-based services include providing notice, requesting consent and implementing safeguards. For research authentication, notice, consent and efficiency. U.S. researchers should seek to follow the FTC's Fair Information Practice Principles.

Clearly, researchers prefer self-regulation to external regulation and Howard encourages researchers to work with him to shape best practices and influence legislation.

MRA First Outlook Conference 2009

Jeffrey Henning — Wed, 04 Nov 2009 17:35:00 GMT

Recaps from a few of the sessions of the Marketing Research Association's First Outlook Conference in San Diego:

Replying to the Voice of the Customer: A Twitter Experiment

Jeffrey Henning — Tue, 03 Nov 2009 15:47:00 GMT

Six weeks ago, as an experiment, I set up a new Twitter account (my main Twitter account is @jhenning) to tweet my personal experiences with products, services and establishments. Since about a third of my tweets would be about local establishments, I sought out and followed about 100 other Twitter users near me; about 20 followed me back.

That done, I then tried to make at least one comment each business day. Each tweet reflected an authentic experience: some were positive comments, some negative, some mixed. I wrote about 30 local, regional and national brands.

My expectation was to do an analysis of brand response by scale of brand and by type of tweet (positive, negative, mixed). Unfortunately, only one brand - a regional brand - ever replied to me. So this makes for a rather boring statistical analysis!

The results shocked me - I rarely tweet about personal brand experiences from @jhenning but the one time I did, the retailer responded to me right away. I had expected a fifth to a third of the brands to respond to me and had hypothesized that regional brands would have the greatest participation rate, as they are big enough to monitor social media and small enough to be early adopters of new technology.

So, if your organization is out there listening on Twitter, it is time to speak up as well. The inaugural survey of the Global Web Index (a syndicated research offering from TrendStream) reported that 22% of its 16,000 panelists said that their perception of a brand is improved if the organization responds to comments in online communities and forums. Sometimes listening to the voice of the customer isn't enough, sometimes acting on the voice of the customer isn't enough: sometimes you have to reply to the voice of the customer.

Perceived Questionnaire Length

Jeffrey Henning — Mon, 02 Nov 2009 15:48:00 GMT

Back in 2002, Mirta Galešic of the University of Zagreb wrote an interesting paper that examined respondents' perception of questionnaire length, "Effects of questionnaire length on response rates: Review of findings and guidelines for future research". If objects in a side-view mirror are closer than they appear, then questionnaires appear to respondents to be longer than they actually are.

Galešic analyzed the relationship between objective and subjective questionnaire length. For objective length, she used the number of questions actually answered (to keep it simple, she treated each question as a question, regardless of its length or type). For subjective length, respondents were asked to rate the questionnaire they had just completed as «too short», «optimal», «somewhat too long» or «absolutely too long» (actual labels were in Croatian, as was the entire questionnaire). Not a single one of the 2,059 respondents answered «too short»!

Galešic writes:

Across all three questionnaire types there was an overall significant, but very small positive correlation between number of questions the respondents answered and their perception of questionnaire length (r=0.11, p<.01). Perceived length was more strongly correlated to the level of interest for the questionnaire topic (r=-.26, p<.01). The less interesting the questionnaire topic was, the longer the questionnaire was perceived to be. Level of interest for the questionnaire topic was not correlated to the number of questions answered (r=.03, p>.05).
Interestingly, however, respondents who had less interest in the topic judged the questionnaire equally long no matter how many questions were answered (the average number of questions answered ranged from 15 to 21 for each of the three surveys).

In the past I've provided six tips for shortening questionnaires. Thanks to this research, here's a seventh: Make the survey interesting to the respondent, and you will shorten the perceived length of the questionnaire.

Survey Trick or Treat: 7 Goblins, Ghosts and Gremlins to Avoid

Brian Koma — Fri, 30 Oct 2009 09:37:00 GMT

Are you spooked by poor survey response rates? Do you get a cold chill when your boss asks what can be done to make your surveys better? Are there skeletons in your survey closet you'd rather avoid? Are you bedeviled by survey bias, poor response rates and bad reporting? By understanding the problems that haunt most surveys, you can enhance the quality of your efforts, dramatically improve the value of your data and ensure high participation rates.

See if you recognize any of these goblins, ghosts and gremlins:

Zombie Surveys - In the movie "Shaun of the Dead," zombies move forward relentlessly under their own power, but have no thinking behind them. Zombie surveys are typified by survey projects that occur year after year because "we've always done it that way." Since good surveys start with good goals, getting rid of zombie surveys means asking two critical questions: Do I really need to conduct this survey to get this data? What action am I going to take with the data I gather? If you can't answer these questions, you've got a zombie to kill.
Frankenstein Surveys - Dr. Frankenstein bolted a monster together out of unrelated parts, and many organizations create surveys the same way. Too often individual departments are asked to contribute questions to a survey, resulting in an out-of-control monster. Avoiding Frankenstein surveys requires you to relentlessly narrow the scope of your survey and focus only on the data you need to make business decisions. Don't stitch together too many survey questions, but use a scalpel to cut out as many questions as possible.
Jekyll & Hyde Surveys - Dr. Jekyll looked perfectly normal, but within minutes of drinking his potion he turned evil. Jekyll & Hyde surveys look normal at the beginning, but quickly turn bad by injecting biased questions or by skewing response scales to summon a pre-ordained result. Never ask questions in such a way that respondents can determine where you stand on any topic. You can avoid drinking Dr. Hyde's potion by striving to write objective questions.
Response Rate Ouija Board - Conjuring up a high response rate requires more than just a dark room and the right incantations. The right ingredients for great response rates are:

1 high quality list of survey invitees
1 tasty survey invitation
6 key points for the survey invitation text
a spoonful of tricks to avoid spam filters and blacklisting
a generous helping of treats and incentives to overcome non-response bias

Vampire Invitations - Bats can reach their destination even in complete darkness, but some of them turn into bloodthirsty vampires. Getting invitations out to potential respondents can involve a similar transformation that can limit your ability to field the survey if you're using non-permission based lists, or more importantly, not complying with the CAN-SPAM Act of 2003. Repelling Vampire Invitations means your e-mail must be viewed as a friendly spirit by following these simple guidelines:

E-mail text contains the physical street address of the sender
Subject line is accurate and does not mislead the invitee
"From" line contains the name of the company or representative
Content includes a valid opt-out or unsubscribe link
Email list has been reduced by removing names on your suppression list

Headless Horseman Reporting - If you're developing mindless reports that get buried and ignored, you're a victim of Headless Horseman Reporting. Since insights are the reason you conducted a survey, you've got to concentrate on survey reports that people will be dying to read. Overcoming this particularly pernicious gremlin means that you mustn't feel compelled to just report on data in the order it was gathered. Call out the most important elements no matter where they were collected in the survey. Make sure your report addresses the Essential Question that inspired the research in the first place.
Silence of the Lambs - Hannibal Lecter may have done despicable things, but he wasn't shy about talking about them ("I ate his liver with some fava beans and a nice Chianti"). You too must talk up your work. Consider developing summary reports, web seminars or blog posts about your survey data, or ultimately, create an online community to discuss results and show people that you're listening. For the greatest return on your survey investment, engage employees with VOC data, then share your results with customers to close the feedback loop and open up greater participation with your next research.

Exercise these best practices to exorcise the goblins, ghosts and gremlins from your research projects!

The Future Consumer: Co-creating the 2020 Kitchen

Jeffrey Henning — Wed, 28 Oct 2009 16:29:00 GMT

Darren Lewis and Koen van der Wal of MetrixLab discussed co-creation at the 2009 ESOMAR Online Research conference. Three emerging themes for product innovation are democratic innovation, Web 2.0 possibilities and active consumers.
Why Co-creation? Consumers make more informed decisions, thanks to unparalleled access to comparative information. Lego provides a great example of co-creation: consumers can build and upload their own ideas for Lego kits and receive a royalty if their idea becomes a product.

Online Qualitative: Research has been more successful at using the Internet for quantitative research than qualitative research. E-groups and bulletin boards require new skills in research moderation; many qualitative researchers argue that much of the value of the focus group dynamic is lost using such methods.

Our Co-Creative Methods: MetrixLab begins with a self-administered depth interview with the results unavailable to other participants, prompting greater candor. The MetrixLab co-creative process engages customers, unleashes creativity and discovers opportunities. Interaction is limited to seeing other participants' contributed ideas. The phases (often space a week or more apart):

Problem analysis and individual idea generation
Idea sharing and enrichment
Evaluation by customer
Evaluation by professional
Selection

Kitchen 2020 Case Study: One hundred and fifty participants in the UK and Netherlands worked to develop the future kitchen. A video introduction to the project provided a more personal description of the project. Participants could create a "moodboard", a bulletin board with images and colors to capture their thinking. Tools channel different creative techniques: pictures and associations, document current frustrations, explore future scenarios, ask "what if?" Participants also wrote descriptions of their moodboards. Themes were the Clean Kitchen, the Green Kitchen, the Flexible Kitchen, the Connected Kitchen and the Automatic Kitchen. The Green Kitchen might incorporate herb gardens and hydroponics, for instance.

Recommendations:

Do's

Use social media to recruit (e.g., cigar lovers were recruited from Facebook)
Involve a diverse group of people
Inspire and motivate people to join your project
Make the process fun, personal and engaging
Share the results and give feedback to customers

Don't's

Don't leave everything open and unstructured
Don't present a challenge that is too broad or too vague
Don't forget to involve the client
Don't judge ideas too fast

Best Practices in Mobile Research

Jeffrey Henning — Wed, 28 Oct 2009 15:32:00 GMT

Chris Ferneyhough and Sonia Bishop of Vision Critical discussed best practices for fielding online surveys to mobile audiences in the ESOMAR Online Research 2009 conference. Chris began by pointing out that mobile Internet adoption outpaces desktop Internet adoption and forecasts that eventually usage of the mobile web will be 10 times usage of the traditional web.

Vision Critical asked respondents to an online survey if they had received their email invitation on their phone: 1.9% had in the US, 1.2% in UK and 3.8% in Canada. Clearly, respondents are already completing online surveys on mobile devices, even though authors have often not taken this into account. Chris mentioned that a panel registration survey for a smart phone vendor didn't actually work for that smart phone, because of the registration form's reliance on JavaScript, which was off by default on the phone: the open-ended "Other (please specify)" box was locked and disabled because it required JavaScript to enable it once the corresponding radio button was clicked. Since client-side scripting is disabled on many phones, the data your survey collected may be wrong.

Researchers need to recognize the fact that online surveys are being completed on mobile devices and need to be optimized for that medium. The wide variety of smartphones at GSMArena.com reveals hundreds of different models with dramatically different market share in different countries. Colors and fonts are implemented differently by different phones and may not be implemented at all on a smart phone: a certain color may render some text unreadable.

Many respondents are unfamiliar with their web browser or alphanumeric entry mode on their phone's keypad. Many respondents are concerned about data costs: some have unlimited data plans, others have pay-as-you-go plans. Many have low data connection speeds.

To do further research on these topics, Vision Critical studied respondents who are smartphone users and are willing to complete questionnaires on their phone. The sample was balanced and weighted on gender and age, with 500 Canadian respondents, 118 US respondents and 107 UK respondents. The survey covered attitudes towards the national economy. No significant statistical differences were found for closed-ended questions on mobile devices vs. desktop devices. For open-ended questions, of course, desktop users were more verbose.

The likelihood to participate in future surveys on mobile phones was greatest for iPhone users (47% were likely to), compared to only 34% of Blackberry users and 23% of all other smartphone users.

Sonia presented back-end mobile research best practices:

Maximize use of the available space
Profile your panel for smartphones, whose email addresses may vary for the phone vs. the desktop
Identify devices and models supported by your data collection software
Manage the process of deploying surveys to mobile panelists

And questionnaire design best practices:

Use simpler question types
Avoid Flash effects and JavaScript validation
Write more concise questions and answer lists to minimize scrolling
Put the Next button "above the fold"
Limit survey length to 10-15 questions (unless heavily incentivized)
Keep it simple: avoid color, grids, images, etc.
Develop for the lowest-common denominator devices

Clearly, survey authors fielding online surveys need to take mobile users into account when developing surveys.

Mobile Interviewing: The Next Frontier of Data Collection

Jeffrey Henning — Wed, 28 Oct 2009 15:01:00 GMT

Steve Lavine of Toluna discussed mobile interviewing at the 2009 ESOMAR Online Research conference, beginning by acknowledging that growth has been slower than expected. While online surveys provide real-time data delivery, such surveys are completed at home or the office: mobile surveys, on the other hand, can now put the survey right at the point of sale.

Mobile interviewing works well when you need immediacy or have to reach younger generations or otherwise hard-to-reach audiences. Mobile interviewing works well for diary studies, where panelists can record impulse purchases as they are being made, eliminating reliance on memory and increasing reporting rates; SMS can be used to send reminders to complete the diary. Camera phones can be used in ethnographic research, as respondents submit audio clips, digital images and even short videos about, for instance, the use of a product. Mobile interviewing can also be used for pharmacological testing.

Handheld devices have long been used for intercept surveys. Mobile tools leverage affordable hardware and work well in developing countries and can provide access to real-time information such as current quota levels.

Good target demographics for mobile interviewing include teens and Generation Y, who are reluctant participants in most other modes of research, but love to text, chat and surf. They love the challenge of the camera phone and eagerly send in images, audio clips and videos; this is true of any income level of teenagers. Much telecommunications research obviously makes sense to do on the mobile phone.
The mobile web provides a rich, extended survey experience with digital images, but many cell phones do not support the web. SMS surveys are discontinuous exchanges of information and are used for simple, text-only polling (a few choose-one or fill-in-the-blank questions), but are accessible to far more people in the U.S. and worldwide than mobile-web surveys. An emerging trend is the downloadable application, which offers rich survey experiences but only to a small installed base; maintaining an access panel of any size or representation is difficult; this works better for a small, well-known group such as employees or partners. IVR surveys fielded to cell phones require the creation and maintenance of an opt-in list for permission contact panelists at the mobile number; invitations are often sent by SMS and the phone IVR survey can be longer, from 5 to 12 minutes in length.

Mobile interviewing provides more rapid response time (2.6 hours) than the response rate of any non-mobile method (5+ hours), often 50% faster. Mobile IVR response time is 3.2 hours vs. 5.1 hours for mobile web surveys. This is a significant benefit for entertainment research. For traditional online surveys, SMS invitations provide a 3.9 hour response time, compared to 6.4 hours for email invitations. Respondents invite by text more closely match the population's age distribution.

In the future, more short-code surveys will be advertised point of sale and upon exiting the store. Expect to see proximity-trigged surveys using GPS, RFID, cell-tower triangulation and other methods, often with panel registration.

How Online Research Communities Work for Consumers Invited to Participate

Jeffrey Henning — Wed, 28 Oct 2009 12:39:00 GMT

Ray Poynter of The Future Place presented "It works for us but does it work for them? How online research communities work for consumers invited to participate" at the ESOMAR Online Research 2009 conference. Researchers and community-platform vendors assume that "communities provide the authentic Voice of the Customer" and "participants love communities". The benefits we assume for participants are that communities make it easier for participants to get their views across, more empowered, with greater convenience, feel the effort is more worthwhile and more enjoyable.

With Lou Rubie of Mars Food and Steven Cierpicki & Daniel Alexander of Colmar Brunton, Poynter did research in Australia, with 1,082 online panel interviews, four focus groups and engaged members of online research communities with discussions, polls and live chat. One hundred percent of respondents to the online survey had done a survey online (whew!), 50% had done a telephone survey (in fact, some had done 10+ telephone surveys), 16% face-to-face interviews, 21% had participated in a focus group (higher than a study of the general population due to the use of panels to recruit participants) and 17% of panel members had participated in online communities (though panelists had broader definitions of communities than MROCs).

	Online	Telephone	Face-to-Face Interviews	Focus Groups	Communities
Always "enjoy participating"	37%	12%	33%	51%	34%
Always "get my views across"	26%	22%	39%	47%	35%
Always "convenient to participate"	44%	12%	29%	34%	39%
Always feel the "return is worth the effort"	20%	10%	28%	42%	30%
Always able to be "completely honest"	71%	49%	54%	59%	57%

Why do only a third of the sample always enjoy participating in communities? From the qualitative research, comments were:

"I am putting in my own views but not hearing back from the client"
One community was too slow to change with too little content
Another community had too much content, with a participant's comments quickly scrolling off the page
The belief that other people participating weren't being honest
People disagreeing with the participant

Communities are less convenient than expected. Why aren't they more convenient? For short-term communities, researchers ask participants to log in four times a day; for other communities, they are asked to upload photos, which is "fun and rewarding but not necessarily convenient!" Researchers give participants a sense of duty: "remember they have a small chance to win a tiny prize!"

Honesty in online surveys comes from the lack of direct observation. In communities, people are conditionally responsive based on what others are saying; it is important to give community members other channels were they can make themselves heard.

Respondents were asked if they were likely to take part in future ORCs (Online Research Communities): of those who had never participated, 96% were interested; of those who participated, 80% were interested in participating in another community.

Caveats: Australia is different from the U.S. and U.K, let alone other countries, so these results may not apply to your market. The results might change in six months to two years, as research communities are evolving so quickly.

For a final report card, Poynter concluded, "Good start but must do better!" Communities are exciting, with wonderful potential "but we are resting on our laurels, we need to do a better job." His five key recommendations:

Ask participants how the experience was for them
Benchmark against "best in class"; don't just compare to other communities but to other experiences and other methodologies.
Make communities as enjoyable as focus groups, even though this is a goal that we may never reach.
Make communities as convenient as online surveys: be more flexible and tailor communities around member preferences.
Make communities as fulfilling as having a real person there: consider doing an offline event where the community gather together or provide visits to the factory to get a tour on how the products are made.

ISO 20252 Standard

Jeffrey Henning — Wed, 28 Oct 2009 10:35:00 GMT

Bill Blyth, Chair of the ISO TC 225 and Global Methods Director of TNS Global described the ISO 20252 standard at the ESOMAR 2009 Online Research conference.

ISO (International Standards Organization) efforts on survey research grew from European trade associations with concerns about the quality of data collection, dating back to the 1970s for standards for face-to-face interviews in the UK and Netherlands. ISO 20252 covers quality at all stages of the survey process: "As some restaurants in London serving offal say, 'It's nose to tail eating!'"

ISO 20252 addresses the quality triangle of Design, Process, User with Fitness in the center. It specifies procedures and documentation and sets minimum levels of validation for key elements. It applies to subcontractors.

While ISO 20252 covers all methods, including online research, the Access Panel Standard - ISO 26362 - published in 2008 provides an alternative to 20252 for specialist panel providers. The 20252 standard is being rewritten to be technology neutral so that it will not need to be rewritten "every time a new gizmo" comes out.

The ISO 20252 has replaced local standards in Australia, Netherlands, Spain and the UK and will do so in France and Sweden in 2010. Italy and Mexico have 20252-inspired local versions. In total, over 250 companies around the world are certified ISO 20252 compliant; the standard does require a compliance audit by a third party.

ISO is encouraging audited compliance to be a contractual/procurement requirement in RFPs. ISO contends that the ISO 20252 saves adopting organizations money by facilitating getting projects right the first time, requiring less rework.

Since online research is the easiest method of research to conduct globally, global standards are needed to ensure quality. ISO 20252 provides a framework and common language for online quality and is a global standard suitable for any size organization in any country.

ARF Online Research Quality Council

Jeffrey Henning — Wed, 28 Oct 2009 10:13:00 GMT

Joel Rubinson, the Chief Research Officer of the ARF, discussed ARF quality initiatives. The ARF ORQC (Online Research Quality Council) was assembled to address the issue of panel quality. The committees developed "Foundations of Quality", a research-on-research project in October and November 2008 conducting 100,000 surveys across 17 U.S. panels, supplemented by phone and mail research.

The research revealed that all seventeen panels could retest their results reliably, but results varied significantly from panel to panel. Switching panels or using a panel undergoing major changes could change the results. The purchase intent for a concept correlates to panelist longevity (panels vary significantly by longevity), but no weighting scheme removed this variation when doing multi-panel sourcing. Rebutting assumptions, taking multiple surveys per month (3 to 10) was actually good for respondent engagement, putting the professional in the term professional respondent, which is usually a disparaging terms. Most panelists were not in it for the money, but those who were provided lower quality answers.

An industry-solutions committee started translating the insights from this research into an action plan, the QeP. The QeP (Quality Enhancement Process) v 1.0 offers templates, definitions, metrics and declarations to bring structure to conversations between buyers and sellers about online panel quality. The QeP is designed as a process to help buyers and sellers meet the shared goal of providing valid and consistent data. QeP is designed to encourage innovation that improves data validity across the online research ecosystem, which includes marketing and research departments on the client side, management consultants, suppliers and subcontractors to those suppliers.

The QeP looks at standards of quality at the panel level, sample/study level and the survey/response-quality level.

At the panel level, panel providers should document recruitment methods, incentive systems, panelist profile, privacy policy, survey QA standards, etc. When panels merge, buyers should be wary of doing trend analysis with results from a predecessor panel.
At the sample/study level, document a consistent sampling plan, promote sample consistency and eliminate duplicate survey taking in any form (deduping households, not just respondents).
At the survey/response-quality level, control for panelist longevity.

The next step for ARF is to pilot test the QeF to ensure that the templates are clear, the documents are workable and the reports are useful and comprehensive. These results will drive future evolution of the QeP.

Update - Joel Rubinson uploaded his presentation to SlideSlide:

Online Research Arf Quality Enhancement Process Esomar Presentation

View more presentations from joel rubinson.