General Mills Moving Qualitative Research Online

Ned Winsborough, manager of consumer networks at General Mills, presented "Accelerating Innovation with Social Networks" at the MRA First Outlook Conference. "We have a mandate at General Mills to move as much of our qualitative research online as possible in the coming months and years. We have been experimenting with this for a year, but we created our consumer networks team this summer and are now scaling it." (Consumer networks is the term that General Mills uses for MROCs.)

General Mills has done 22 community projects since last spring. Why online communities? "Online consumer communities meet the needs of consumers, brand teams and agencies with busy lives. They allow you to innovate with consumers better, faster, and cheaper." With communities, General Mills is able to engage in iterative building of concepts: "We listen, we build; we listen, we tweak. This can be done very quickly, with a lot of flexibility to the method." Community research allows for faster speed to market. For one project, General Mills did six months of work in six weeks. Compared to other qualitative methods, communities are less expensive. "There is a fixed cost for setting up the communities, which can be very significant, but the incremental cost of doing extra weeks, extra moderation, is very low."

As a result of General Mills' 22 projects, they have made changes to their approach to community research:

• Focus on Discovery - The General Mills innovation model uses three steps: Discover, Build, Launch. The communities are great for Discovery but less suited for the Build phase. In the Discovery phase, community research always works, according to Ned, whether the project is big or small, whether the tolerance for risk is high or low. In the Build phase, small projects can be supported with community research but larger projects require traditional quantitative research. For future community research, "we are focusing on Discovery."
• Smaller Communities - Early communities were larger (for example, 225 participants), but that produced too much information to quickly and easily analyze. "Now we work with communities of 30 to 50 people (more if we have subgroups we want to investigate). With fewer members, we really get to know them as individuals, and we can probe better."
• Shorter Duration Communities - General Mills has moved from a permanent online community to project-based communities that last for six to eight weeks. "This is a different model than creating one ongoing community. We have some experience with that type of community: we had done that in the past but found it wasn't cost effective." The ongoing moderation activities can be significant, yet "it is rare that we have things that we need to do every week."
• Larger Incentives - Members to an early community were offered $50 for six weeks participation and a chance to win some modest prizes. Current incentives tend to run $40-50 per week.
• Geographically Centered - For one of its first project communities, General Mills invited seven local participants to come to their facility for shelf tests and project packaging tests. Now, General Mills "uses focus group facilities to recruit members, so that we can do selective face to face research."

Ned has heard everything from "traditional research is dead" to skepticism about the value of online community research. "The truth is in the middle," he said. "It has a place, and we need to approach it like any other new technology. What questions can it answer? What objectives can it meet? What objectives can't it meet? Where can it fit in an array of methods? It certainly doesn't obsolete core quantitative methods but it has powerful potential to transform qualitative research as we know it."

MROC Case Study from ABC Studios

Karen Manne, VP of research with Disney, presented "Journey inside the ABC Studios Advisory Panel" at the MRA First Outlook Conference. ABC Studios is the production company for the Disney television group, producing shows such as Brothers & Sisters, Desperate Housewives and Grey's Anatomy, as well as Castle, FlashForward, Legend of the Seeker and Lost, among others. "We started building this community three years ago," said Karen. "ASAP (the ABC Studios Advisory Panel) was the first online community at Disney and the first for program planning in the TV industry."

The community currently has 1,900 members, who are each heavy viewers of at least two ABC Studios shows and who are opinion leaders: people who are passionate about television and regularly talk to their families and friends about the shows they watch.  Membership fluctuates, as members who don't log in for at least three months are purged occasionally; a purge six months ago reduced membership to 1,400. Unfortunately, the panel is not gender balanced: 86% of the members are female; Karen recently went to ComicCon to unsuccessfully recruit more men for the panel, but most men only watch one of the ABC Studios shows.

Members are not given monetary incentives at all, but participate because they want to have a hand in shaping TV programming. They are sometimes given digital access to TV shows and also see sneak peeks of shows.

For a time, members could refer friends to the community, but Karen has stopped that practice. Too many referrals were skewing some of the research, as members invited others with identical views on particular characters or aspects of the shows.

Here are some anecdotes about the community by show:

• Brothers & Sisters - The producers were interested in viewer opinions of several of the male characters; rather than tip their hand into which characters they were most interested in, they did a general study of all the male characters on the show.
• Castle - As an entertainment company, ABC Studios is able to provide unusual rewards to its members. Three heavy contributors to the site were invited to a book signing of Heat Wave, a real book marketed as if it were written by the fictional Richard Castle (played by Nathan Fillion, of Firefly). These community members were given the VIP treatment and were photographed meeting Fillion; this was then publicized in the community.
• Grey's Anatomy - One problem Karen has experienced is the occasional leak of sensitive information from the panel. A poll about the character Lexie was released to the public, for instance. As a result, for some sensitive polls, respondents are no longer shown the results. ("Polls are a favorite of members, since they're a quick and easy way to provide feedback.")
• Scrubs - ABC bought the rights to this NBC show and is relaunching it with an altered premise this season. The season premiere was uploaded for community members, who were encouraged to watch the whole episode and provide feedback on the significant changes to the story. Initial viewer reaction was positive.
• Ugly Betty - After three years of working at the fashion magazine, Betty is finally getting a makeover, and ASAP members reviewed seven possible new looks. Through leaks from the community, this lead to fevered coverage in blogs (Perez Hilton: De Uglifying Betty) and the entertainment press, and finally, even, the Wall Street Journal ("Making Ugly Betty Prettier: To gauge viewer reaction, ABC turns to online focus groups to test its star").

For ABC Studios, the benefits of ASAP are many:

• Provides easy access to consumers
• Yields quick feedback on insights and attitudes
• "Spontaneity allows for flexibility"
• Targeted research
• Viral marketing

Many research projects are quite small and targeted, leading to shorter, more focused questionnaires. Karen has done literally a 1000 projects in the community.

Karen said, "I love my community but it is not all puppies and rainbows - it takes a lot of work." Some of the cons:

• Unable to verify that members aren't reporters or competitors
• Requires ongoing investment of time and money to recruit new members
• Leaks of sensitive material to the entertainment blogs and press
• Piracy of episodes posted within the community
• Qualitative data is voluminous and time consuming to analyze
• Busy work - during off-production times (such as the summer) need to have community activities to keep members engaged for when they are really needed
• Victim of success - get pushed for rapid turnaround because executives realize the community enables it

For all its cons, Karen said the benefits outweigh the challenges. "Online communities are the hot new ticket in market research."

From Bedrock to MROC: Member Activities beyond Discussions

Jane Mount, PRC, an executive vice president with Digital Research, presented "From Bedrock to MROC: An Evolution in Qualitative Research Practices" at the 2009 MRA First Outlook Conference. Jane began by describing how quantitative methods over the past sixty years have evolved from door-to-door techniques, to direct mail, to telephone and to online research, while qualitative research has remained with focus groups during this time period. Even today online focus groups represent less than 10% of the qualitative market. MROCs, however, represent a substantial shift in qualitative research: "a shift from asking questions to get reactive consumer feedback, to listening to dialogue to get proactive consumer insight."

Jane provided an excellent introduction to the topic of MROCs, covering familiar ground. (If you're new to MROCs, see my past posts on MROC = Market Research Online Community, Focus Groups vs. Online Communities and Social Networks vs. Online Communities vs. Panels.)

Jane presented the MROC market as a continuum ranging from full-service suppliers such as her firm, Digital Research, on one side to technology-only suppliers such as Vovici on the other side. [She showed a couple of her competitors, and a couple Vovici competitors, all of whom I have happily omitted from my recreation of her slide!]

Frequently researchers think of MROCs as simply "listening posts", a place to facilitate discussions and eavesdrop on conversations, but - as a full-service supplier - Jane talked about the many other research activities that MROC members can do:

• Static ethnography - Upload photos from their personal life.
• Representational images - Upload clip art or a photo that represents a topic. For a sensitive topic like body image, have them submit these privately. For a fun topic like perceptions of their in-laws, have them upload the image for all to see and comment on (one member uploaded a picture of a cactus with the caption "They're prickly" to describe their in-laws, inspiring a comment thread from others who agreed).
• Idea banks - Submit ideas to a shared database where they can rate them.
• Insight games - Play word-association exercises and MadLibs-style sentence completion games.
• Cartoon captions - Write a caption for a cartoon.
• Personal diaries - Record daily activities, providing a richer narrative than possible through a one-time survey; for instance, revealing how members struggle with dieting on a day-to-day basis.
• Collages - Assemble collages that represent the topic being researched.
• Fun polls & quizzes - Answer entertaining questions like "If the economy was a candy bar, which of these candy bars would it be?"
• Team activities - Do planned exercises with others. The research team segments users upon registration and then plans team activities where each team represents a different segmentation. For instance, DRI did a traditional quantitative study that produced six segments of consumer buying behavior, then invited those respondents into the community and recorded their segmentation.

Jane said that issues suitable for research with MROCs include "ideation; testing social media strategy; trend spotting; early stage evaluation of branding, packaging, ads; exploring attitudes and behaviors; directional insights when time is critical; and testing suitable language for a target." To her mind, MROCs are a very cost-effective method for qualitative research that is gaining in popularity because they are fast, provide ongoing insight generation, are highly creative, and their tech-intensiveness matches with respondent lifestyles today.

MROCs are now part of "a modern Stone Age family" of qualitative tools.

Photo credit:  © 2008 Mike Hill

Multicultural Market Research: Eight Make-or-Break Rules

David R. Morse, president of New American Dimensions, discussed his new book Multicultural Intelligence: Eight Make-or-Break Rules for Marketing to Race, Ethnicity, and Sexual Orientation at the MRA First Outlook Conference. He argued that all U.S. researchers need to develop competency in multicultural research, given shifts such as these in American demographics: 

• The percentage of foreign-born population in the United States has increased to 14%, a level not seen since 1910.
• Hispanic immigration is at an all-time high, and Hispanics will make up 20% of Americans by 2050.
• By 2044, white Americans will be a minority according to projections from the U.S. Census department.

Here are David's eight rules:

1. Boost your multicultural competency. The U.S. Census is very detailed and useful for developing a detailed demographic understanding (though it lacks demographics by sexual orientation). When doing focus groups or qualitative research, make sure to have a moderator or interviewer of the race or sexual orientation being studied.
2. Divide and conquer. Don't accept the stereotypes ("Hispanics are brand loyal") but segment this population to truly understand how it relates to your market. The Current Population Survey is one of the few resources that lets you segment population by generation. Foreign born (or first-generation Americans) are very different from second-generation Americans, who generally speak English as well as their parents' native language; second-generation Americans are different again from the third-generation Americans, who typically speak only English.
3. Don't trust the experts. The accepted wisdom is often full of persistent and incorrect "truths" and urban legends, especially relating to translation. Do your homework.
4. Don't let the joke be on you. Tread carefully when doing multicultural marketing with humor; what is funny differs dramatically by culture.
5. Don't get lost in the translation. Given the prevalence of translation errors, make certain to backtranslate the questionnaire. David once was surprised to see in a survey that 100% of Spanish-speaking Hispanics disagreed that "the Internet is color blind". Rechecking the Spanish translation, he found that it has been translated to mean "the Internet has red-blue-green color blindness".
6. Push their buttons. Find the cultural cues that people resonate with; use your research to determine what those are for your market.
7. Market on a wink and a prayer. In a mainstream ad, insert a subtle cultural reference. The mainstream will miss it, but the targeted culture will appreciate it.
8. Make up; don't cover up. Watch the watchdogs. Many segments have advocacy and anti-discrimination  groups; if your marketing runs afoul of them, stop the campaign at once and apologize.

America is not one culture, and savvy researchers realize this. "We are not a color-blind society," said David. "We live in different worlds. We talk differently. We listen to different kinds of music. We worship differently." Follow David's eight rules to improve your organization's marketing and market research.

Market Research at Microsoft: Evolution of the MR Department

At the MRA First Outlook Conference in San Diego, Reed Cundiff, senior director of central market research for Microsoft (and formerly an analyst with the Yankee Group), discussed how market research at the company has evolved and is evolving.

Prior to the creation of the Central Market Research Insights team, researchers had existed alone or in pairs in many different departments across the company. Job descriptions varied significantly; researchers had no career path within Microsoft.
Four years ago, the CMRI had just eight staff. It peaked at about 102 people in June, before being brought down by layoff to 97 staff.  The upside of the downside, as it were, is that it has further accelerated the centralization of market research: departments that in the past were funding their own research are now turning to the central group instead. "We see that a lot of ad hoc research budgets have been cut; that is good for us, as there were many projects done outside of our research group." As a result, the organization is eliminating redundant and superfluous research expenditures.

The vision for the central research group is "to be a driving force behind Microsoft's business and product strategy" with the mission of delivering "strategic, fact-based insights that drive Microsoft's most essential business decisions."

The organization spends $80M to $110M annually on external research: the technology sector's largest research budget, according to Reed. Projects range from doing a market opportunity analysis for a v1 health care product to conducting a customer satisfaction survey with 100,000 respondents across 86 countries. CMRI devotes 3.5 FTEs to its research vendor management program, where they develop the preferred vendor list and do biannual reviews of the vendors (and ask the vendors to review the research managers they interact with). The result has been better use of outside vendors and consistent improvement.  Before the new process, Microsoft would "run a pilot with a vendor, fall in love with that vendor, bury that vendor, then never do business with that vendor again!"

CMRI's strategy has three key components: to "deliver integrated insights" (primary research with market analysis), to "be a trusted advisor" and to "display business acumen". As a result, what Microsoft wants and expects from its internal researchers is changing [worth a blog post of its own!]: researchers need to be more consultative and need to specialize in a few focus areas.

In its research on research, the CMRI has adopted Six Sigma. "In the past fiscal year, we went through the Six Sigma process and we are reducing the number of defects study by study by study. We averaged 12 defects per final report in a six-month period to 3 the next period down to 2 most recently."

"We need to seize the opportunity. We are moving through a lot of challenges but to drive a fact-based culture, the timing has never been better for us." In his concluding remarks, Reed said, "We spend millions of dollars that affects hundreds of millions of dollars in marketing spend that affects billions of dollars in revenue. We have to get it right."

Market Research Regulation in 2010

At the MRA First Outlook Conference, Howard Feinberg (@hfienberg), director of government affairs of the Market Research Association, presented "Regulation Over the Horizon: Emerging Research Technologies & Modes and the Legal & Privacy Hurdles". Howard covered six broad trends that have implications for the future of market-research regulation.

• Online behavioral tracking - FTC proposes self-regulatory standards, which means "you need to regulate yourselves, or we will do it for you". Facebook recently settled a lawsuit about Beacon, which was considered to be in violation of established, offline privacy laws. Public opinion about behavioral tracking is mixed: users prefer free web content, paid for by ads, to subscription websites and are aware that information about their web browsing is collected, but are disturbed by the "creepiness" factor when they are unsure about what information is collected or how it is disseminated. In a recent survey, 92% of respondents felt that there should be a law requiring websites and ad servers to delete all information about them upon request.
• Location and behavioral data - A little appreciated downside of our connectivity is that we've given up locational privacy, as we are tracked by our cellphones and GPS devices in our cars (if you know someone's home and work address you can identify their GPS data stream); sites like Google Latitude and Loopt provide information about where opt-in users are in real time. Locational data is not just about where you go, since those locations demonstrate associations - "political, religious, amicable and amorous, to name only a few," according to a New York State Court of Appeals ruling in May. Even with GPS off, phones can triangulate location from cell towers. Other developments that provide the MR industry great opportunities for observational research from data aggregation:
• Car insurance companies in California were recently given approval to charge rates based on travel patterns and mileage.
• Feinberg joked about RFID tagging of research participants, pointing out that RFID-tagged passports are now used at U.S. border crossings.
• A smart grid for power management of networked appliances and HVAC systems could provide insight into intimate consumer usage of appliances.
• UK billboards photograph license-plate numbers to look up the make and model of the vehicle, showing an ad for the exact motor oil required by that vehicle.
  
• Social media - Scraping social media is limited by concerns about data quality, which prompts questions about who these people really are. From an ethical standpoint, are users aware that you are watching, listening, reading, analyzing their output? Teens and tweens are often naïve about the privacy of their data. New laws are regulating social networks, without defining social networks; this could have implications for researchers, if online focus groups and online research communities become classified and regulated as social networks. [See Social Networks vs. Online Communities vs. Panels for my definition.]
• Respondent authentication -One survey found that web users were open to authentication by trusted vendors. No U.S. law currently governs digital fingerprinting, which is used by many panels for respondent authentication. In Europe, an IP address may constitute protected PII (Personally Identifiable Information).
• Cloud computing - The provision of data through servers accessible over the Internet raises risks for data security; Howard advised researchers to make certain to download and backup data stored on such systems. How do international data transfer laws relate to cloud computing? This is yet to be decided.
• Ethical and legal initiatives - For self-regulation of the industry, researchers are urged to provide transparency and consumer control over data, with limits on data retention. A standard practice, widely adopted, is get consumer consent whenever a privacy policy is changed. An additional practice is to get consent for using "sensitive" data. Best practices for location-based services include providing notice, requesting consent and implementing safeguards. For research authentication, notice, consent and efficiency. U.S. researchers should seek to follow the FTC's Fair Information Practice Principles.

Clearly, researchers prefer self-regulation to external regulation and Howard encourages researchers to work with him to shape best practices and influence legislation.

MRA First Outlook Conference 2009

Recaps from a few of the sessions of the Marketing Research Association's First Outlook Conference in San Diego:

• Market Research Regulation in 2010
• Market Research at Microsoft: Evolution of the MR Department  
• Multicultural Market Research: Eight Make or Break Rules  
• From Bedrock to MROC: Member Activities beyond Discussions
• MROC Case Study from ABC Studios
• General Mills Moving Qualitative Research Online
• More to come Friday!

Replying to the Voice of the Customer: A Twitter Experiment

Six weeks ago, as an experiment, I set up a new Twitter account (my main Twitter account is @jhenning) to tweet my personal experiences with products, services and establishments. Since about a third of my tweets would be about local establishments, I sought out and followed about 100 other Twitter users near me; about 20 followed me back.

That done, I then tried to make at least one comment each business day. Each tweet reflected an authentic experience: some were positive comments, some negative, some mixed. I wrote about 30 local, regional and national brands.

My expectation was to do an analysis of brand response by scale of brand and by type of tweet (positive, negative, mixed). Unfortunately, only one brand - a regional brand - ever replied to me. So this makes for a rather boring statistical analysis!

The results shocked me - I rarely tweet about personal brand experiences from @jhenning but the one time I did, the retailer responded to me right away. I had expected a fifth to a third of the brands to respond to me and had hypothesized that regional brands would have the greatest participation rate, as they are big enough to monitor social media and small enough to be early adopters of new technology.

So, if your organization is out there listening on Twitter, it is time to speak up as well.  The inaugural survey of the Global Web Index (a syndicated research offering from TrendStream) reported that 22% of its 16,000 panelists said that their perception of a brand is improved if the organization responds to comments in online communities and forums. Sometimes listening to the voice of the customer isn't enough, sometimes acting on the voice of the customer isn't enough: sometimes you have to reply to the voice of the customer.

Perceived Questionnaire Length

Back in 2002, Mirta Galešic of the University of Zagreb wrote an interesting paper that examined respondents' perception of questionnaire length, "Effects of questionnaire length on response rates: Review of findings and guidelines for future research". If objects in a side-view mirror are closer than they appear, then questionnaires appear to respondents to be longer than they actually are.

Galešic analyzed the relationship between objective and subjective questionnaire length. For objective length, she used the number of questions actually answered (to keep it simple, she treated each question as a question, regardless of its length or type). For subjective length, respondents were asked to rate the questionnaire they had just completed as «too short», «optimal», «somewhat too long» or «absolutely too long» (actual labels were in Croatian, as was the entire questionnaire). Not a single one of the 2,059 respondents answered «too short»!

Galešic writes:

Across all three questionnaire types there was an overall significant, but very small positive correlation between number of questions the respondents answered and their perception of questionnaire length (r=0.11, p<.01). Perceived length was more strongly correlated to the level of interest for the questionnaire topic (r=-.26, p<.01). The less interesting the questionnaire topic was, the longer the questionnaire was perceived to be. Level of interest for the questionnaire topic was not correlated to the number of questions answered (r=.03, p>.05).
Interestingly, however, respondents who had less interest in the topic judged the questionnaire equally long no matter how many questions were answered (the average number of questions answered ranged from 15 to 21 for each of the three surveys).

 

In the past I've provided six tips for shortening questionnaires. Thanks to this research, here's a seventh: Make the survey interesting to the respondent, and you will shorten the perceived length of the questionnaire.

Survey Trick or Treat: 7 Goblins, Ghosts and Gremlins to Avoid

Are you spooked by poor survey response rates? Do you get a cold chill when your boss asks what can be done to make your surveys better? Are there skeletons in your survey closet you'd rather avoid? Are you bedeviled by survey bias, poor response rates and bad reporting? By understanding the problems that haunt most surveys, you can enhance the quality of your efforts, dramatically improve the value of your data and ensure high participation rates.

See if you recognize any of these goblins, ghosts and gremlins:

1. Zombie Surveys - In the movie "Shaun of the Dead," zombies move forward relentlessly under their own power, but have no thinking behind them. Zombie surveys are typified by survey projects that occur year after year because "we've always done it that way." Since good surveys start with good goals, getting rid of zombie surveys means asking two critical questions: Do I really need to conduct this survey to get this data? What action am I going to take with the data I gather? If you can't answer these questions, you've got a zombie to kill.
2. Frankenstein Surveys - Dr. Frankenstein bolted a monster together out of unrelated parts, and many organizations create surveys the same way. Too often individual departments are asked to contribute questions to a survey, resulting in an out-of-control monster. Avoiding Frankenstein surveys requires you to relentlessly narrow the scope of your survey and focus only on the data you need to make business decisions. Don't stitch together too many survey questions, but use a scalpel to cut out as many questions as possible.
3. Jekyll & Hyde Surveys - Dr. Jekyll looked perfectly normal, but within minutes of drinking his potion he turned evil. Jekyll & Hyde surveys look normal at the beginning, but quickly turn bad by injecting biased questions or by skewing response scales to summon a pre-ordained result. Never ask questions in such a way that respondents can determine where you stand on any topic. You can avoid drinking Dr. Hyde's potion by striving to write objective questions.
4. Response Rate Ouija Board - Conjuring up a high response rate requires more than just a dark room and the right incantations. The right ingredients for great response rates are:
• 1 high quality list of survey invitees
• 1 tasty survey invitation
• 6 key points for the survey invitation text
• a spoonful of tricks to avoid spam filters and blacklisting
• a generous helping of treats and incentives to overcome non-response bias 
5. Vampire Invitations - Bats can reach their destination even in complete darkness, but some of them turn into bloodthirsty vampires. Getting invitations out to potential respondents can involve a similar transformation that can limit your ability to field the survey if you're using non-permission based lists, or more importantly, not complying with the CAN-SPAM Act of 2003.  Repelling Vampire Invitations means your e-mail must be viewed as a friendly spirit by following these simple guidelines:
• E-mail text contains the physical street address of the sender
• Subject line is accurate and does not mislead the invitee
• "From" line contains the name of the company or representative
• Content includes a valid opt-out or unsubscribe link
• Email list has been reduced by removing names on your suppression list
6. Headless Horseman Reporting - If you're developing mindless reports that get buried and ignored, you're a victim of Headless Horseman Reporting. Since insights are the reason you conducted a survey, you've got to concentrate on survey reports that people will be dying to read. Overcoming this particularly pernicious gremlin means that you mustn't feel compelled to just report on data in the order it was gathered. Call out the most important elements no matter where they were collected in the survey. Make sure your report addresses the Essential Question that inspired the research in the first place.
7. Silence of the Lambs - Hannibal Lecter may have done despicable things, but he wasn't shy about talking about them ("I ate his liver with some fava beans and a nice Chianti"). You too must talk up your work. Consider developing summary reports, web seminars or blog posts about your survey data, or ultimately, create an online community to discuss results and show people that you're listening. For the greatest return on your survey investment, engage employees with VOC data, then share your results with customers to close the feedback loop and open up greater participation with your next research.

Exercise these best practices to exorcise the goblins, ghosts and gremlins from your research projects!