Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
|
 RSS Feed
Posted by Jeffrey Henning on Fri, Nov 06, 2009
 Ned Winsborough, manager of consumer networks at General Mills, presented "Accelerating Innovation with Social Networks" at the MRA First Outlook Conference. "We have a mandate at General Mills to move as much of our qualitative research online as possible in the coming months and years. We have been experimenting with this for a year, but we created our consumer networks team this summer and are now scaling it." (Consumer networks is the term that General Mills uses for MROCs.)
General Mills has done 22 community projects since last spring. Why online communities? "Online consumer communities meet the needs of consumers, brand teams and agencies with busy lives. They allow you to innovate with consumers better, faster, and cheaper." With communities, General Mills is able to engage in iterative building of concepts: "We listen, we build; we listen, we tweak. This can be done very quickly, with a lot of flexibility to the method." Community research allows for faster speed to market. For one project, General Mills did six months of work in six weeks. Compared to other qualitative methods, communities are less expensive. "There is a fixed cost for setting up the communities, which can be very significant, but the incremental cost of doing extra weeks, extra moderation, is very low."
As a result of General Mills' 22 projects, they have made changes to their approach to community research:
- Focus on Discovery - The General Mills innovation model uses three steps: Discover, Build, Launch. The communities are great for Discovery but less suited for the Build phase. In the Discovery phase, community research always works, according to Ned, whether the project is big or small, whether the tolerance for risk is high or low. In the Build phase, small projects can be supported with community research but larger projects require traditional quantitative research. For future community research, "we are focusing on Discovery."
- Smaller Communities - Early communities were larger (for example, 225 participants), but that produced too much information to quickly and easily analyze. "Now we work with communities of 30 to 50 people (more if we have subgroups we want to investigate). With fewer members, we really get to know them as individuals, and we can probe better."
- Shorter Duration Communities - General Mills has moved from a permanent online community to project-based communities that last for six to eight weeks. "This is a different model than creating one ongoing community. We have some experience with that type of community: we had done that in the past but found it wasn't cost effective." The ongoing moderation activities can be significant, yet "it is rare that we have things that we need to do every week."
- Larger Incentives - Members to an early community were offered $50 for six weeks participation and a chance to win some modest prizes. Current incentives tend to run $40-50 per week.
- Geographically Centered - For one of its first project communities, General Mills invited seven local participants to come to their facility for shelf tests and project packaging tests. Now, General Mills "uses focus group facilities to recruit members, so that we can do selective face to face research."
Ned has heard everything from "traditional research is dead" to skepticism about the value of online community research. "The truth is in the middle," he said. "It has a place, and we need to approach it like any other new technology. What questions can it answer? What objectives can it meet? What objectives can't it meet? Where can it fit in an array of methods? It certainly doesn't obsolete core quantitative methods but it has powerful potential to transform qualitative research as we know it."
Posted by Jeffrey Henning on Thu, Nov 05, 2009
 Karen Manne, VP of research with Disney, presented "Journey inside the ABC Studios Advisory Panel" at the MRA First Outlook Conference. ABC Studios is the production company for the Disney television group, producing shows such as Brothers & Sisters, Desperate Housewives and Grey's Anatomy, as well as Castle, FlashForward, Legend of the Seeker and Lost, among others. "We started building this community three years ago," said Karen. "ASAP (the ABC Studios Advisory Panel) was the first online community at Disney and the first for program planning in the TV industry."
The community currently has 1,900 members, who are each heavy viewers of at least two ABC Studios shows and who are opinion leaders: people who are passionate about television and regularly talk to their families and friends about the shows they watch. Membership fluctuates, as members who don't log in for at least three months are purged occasionally; a purge six months ago reduced membership to 1,400. Unfortunately, the panel is not gender balanced: 86% of the members are female; Karen recently went to ComicCon to unsuccessfully recruit more men for the panel, but most men only watch one of the ABC Studios shows.
Members are not given monetary incentives at all, but participate because they want to have a hand in shaping TV programming. They are sometimes given digital access to TV shows and also see sneak peeks of shows.
For a time, members could refer friends to the community, but Karen has stopped that practice. Too many referrals were skewing some of the research, as members invited others with identical views on particular characters or aspects of the shows.
Here are some anecdotes about the community by show:
- Brothers & Sisters - The producers were interested in viewer opinions of several of the male characters; rather than tip their hand into which characters they were most interested in, they did a general study of all the male characters on the show.
- Castle - As an entertainment company, ABC Studios is able to provide unusual rewards to its members. Three heavy contributors to the site were invited to a book signing of Heat Wave, a real book marketed as if it were written by the fictional Richard Castle (played by Nathan Fillion, of Firefly). These community members were given the VIP treatment and were photographed meeting Fillion; this was then publicized in the community.
- Grey's Anatomy - One problem Karen has experienced is the occasional leak of sensitive information from the panel. A poll about the character Lexie was released to the public, for instance. As a result, for some sensitive polls, respondents are no longer shown the results. ("Polls are a favorite of members, since they're a quick and easy way to provide feedback.")
- Scrubs - ABC bought the rights to this NBC show and is relaunching it with an altered premise this season. The season premiere was uploaded for community members, who were encouraged to watch the whole episode and provide feedback on the significant changes to the story. Initial viewer reaction was positive.
- Ugly Betty - After three years of working at the fashion magazine, Betty is finally getting a makeover, and ASAP members reviewed seven possible new looks. Through leaks from the community, this lead to fevered coverage in blogs (Perez Hilton: De Uglifying Betty) and the entertainment press, and finally, even, the Wall Street Journal ("Making Ugly Betty Prettier: To gauge viewer reaction, ABC turns to online focus groups to test its star").
For ABC Studios, the benefits of ASAP are many:
- Provides easy access to consumers
- Yields quick feedback on insights and attitudes
- "Spontaneity allows for flexibility"
- Targeted research
- Viral marketing
Many research projects are quite small and targeted, leading to shorter, more focused questionnaires. Karen has done literally a 1000 projects in the community.
Karen said, "I love my community but it is not all puppies and rainbows - it takes a lot of work." Some of the cons:
- Unable to verify that members aren't reporters or competitors
- Requires ongoing investment of time and money to recruit new members
- Leaks of sensitive material to the entertainment blogs and press
- Piracy of episodes posted within the community
- Qualitative data is voluminous and time consuming to analyze
- Busy work - during off-production times (such as the summer) need to have community activities to keep members engaged for when they are really needed
- Victim of success - get pushed for rapid turnaround because executives realize the community enables it
For all its cons, Karen said the benefits outweigh the challenges. "Online communities are the hot new ticket in market research."
Posted by Jeffrey Henning on Thu, Nov 05, 2009
 Jane Mount, PRC, an executive vice president with Digital Research, presented "From Bedrock to MROC: An Evolution in Qualitative Research Practices" at the 2009 MRA First Outlook Conference. Jane began by describing how quantitative methods over the past sixty years have evolved from door-to-door techniques, to direct mail, to telephone and to online research, while qualitative research has remained with focus groups during this time period. Even today online focus groups represent less than 10% of the qualitative market. MROCs, however, represent a substantial shift in qualitative research: "a shift from asking questions to get reactive consumer feedback, to listening to dialogue to get proactive consumer insight."
Jane provided an excellent introduction to the topic of MROCs, covering familiar ground. (If you're new to MROCs, see my past posts on MROC = Market Research Online Community, Focus Groups vs. Online Communities and Social Networks vs. Online Communities vs. Panels.)
Jane presented the MROC market as a continuum ranging from full-service suppliers such as her firm, Digital Research, on one side to technology-only suppliers such as Vovici on the other side. [She showed a couple of her competitors, and a couple Vovici competitors, all of whom I have happily omitted from my recreation of her slide!]
Frequently researchers think of MROCs as simply "listening posts", a place to facilitate discussions and eavesdrop on conversations, but - as a full-service supplier - Jane talked about the many other research activities that MROC members can do:
- Static ethnography - Upload photos from their personal life.
- Representational images - Upload clip art or a photo that represents a topic. For a sensitive topic like body image, have them submit these privately. For a fun topic like perceptions of their in-laws, have them upload the image for all to see and comment on (one member uploaded a picture of a cactus with the caption "They're prickly" to describe their in-laws, inspiring a comment thread from others who agreed).
- Idea banks - Submit ideas to a shared database where they can rate them.
- Insight games - Play word-association exercises and MadLibs-style sentence completion games.
- Cartoon captions - Write a caption for a cartoon.
- Personal diaries - Record daily activities, providing a richer narrative than possible through a one-time survey; for instance, revealing how members struggle with dieting on a day-to-day basis.
- Collages - Assemble collages that represent the topic being researched.
- Fun polls & quizzes - Answer entertaining questions like "If the economy was a candy bar, which of these candy bars would it be?"
- Team activities - Do planned exercises with others. The research team segments users upon registration and then plans team activities where each team represents a different segmentation. For instance, DRI did a traditional quantitative study that produced six segments of consumer buying behavior, then invited those respondents into the community and recorded their segmentation.
Jane said that issues suitable for research with MROCs include "ideation; testing social media strategy; trend spotting; early stage evaluation of branding, packaging, ads; exploring attitudes and behaviors; directional insights when time is critical; and testing suitable language for a target." To her mind, MROCs are a very cost-effective method for qualitative research that is gaining in popularity because they are fast, provide ongoing insight generation, are highly creative, and their tech-intensiveness matches with respondent lifestyles today.
MROCs are now part of "a modern Stone Age family" of qualitative tools.
Photo credit: © 2008 Mike Hill
Posted by Jeffrey Henning on Wed, Oct 28, 2009
 Ray Poynter of The Future Place presented "It works for us but does it work for them? How online research communities work for consumers invited to participate" at the ESOMAR Online Research 2009 conference. Researchers and community-platform vendors assume that "communities provide the authentic Voice of the Customer" and "participants love communities". The benefits we assume for participants are that communities make it easier for participants to get their views across, more empowered, with greater convenience, feel the effort is more worthwhile and more enjoyable.
With Lou Rubie of Mars Food and Steven Cierpicki & Daniel Alexander of Colmar Brunton, Poynter did research in Australia, with 1,082 online panel interviews, four focus groups and engaged members of online research communities with discussions, polls and live chat. One hundred percent of respondents to the online survey had done a survey online (whew!), 50% had done a telephone survey (in fact, some had done 10+ telephone surveys), 16% face-to-face interviews, 21% had participated in a focus group (higher than a study of the general population due to the use of panels to recruit participants) and 17% of panel members had participated in online communities (though panelists had broader definitions of communities than MROCs).
|
|
Online |
Telephone |
Face-to-Face Interviews |
Focus Groups |
Communities |
|
Always "enjoy participating" |
37% |
12% |
33% |
51% |
34% |
|
Always "get my views across" |
26% |
22% |
39% |
47% |
35% |
|
Always "convenient to participate" |
44% |
12% |
29% |
34% |
39% |
|
Always feel the "return is worth the effort" |
20% |
10% |
28% |
42% |
30% |
|
Always able to be "completely honest" |
71% |
49% |
54% |
59% |
57% |
Why do only a third of the sample always enjoy participating in communities? From the qualitative research, comments were:
- "I am putting in my own views but not hearing back from the client"
- One community was too slow to change with too little content
- Another community had too much content, with a participant's comments quickly scrolling off the page
- The belief that other people participating weren't being honest
- People disagreeing with the participant
Communities are less convenient than expected. Why aren't they more convenient? For short-term communities, researchers ask participants to log in four times a day; for other communities, they are asked to upload photos, which is "fun and rewarding but not necessarily convenient!" Researchers give participants a sense of duty: "remember they have a small chance to win a tiny prize!"
Honesty in online surveys comes from the lack of direct observation. In communities, people are conditionally responsive based on what others are saying; it is important to give community members other channels were they can make themselves heard.
Respondents were asked if they were likely to take part in future ORCs (Online Research Communities): of those who had never participated, 96% were interested; of those who participated, 80% were interested in participating in another community.
Caveats: Australia is different from the U.S. and U.K, let alone other countries, so these results may not apply to your market. The results might change in six months to two years, as research communities are evolving so quickly.
For a final report card, Poynter concluded, "Good start but must do better!" Communities are exciting, with wonderful potential "but we are resting on our laurels, we need to do a better job." His five key recommendations:
- Ask participants how the experience was for them
- Benchmark against "best in class"; don't just compare to other communities but to other experiences and other methodologies.
- Make communities as enjoyable as focus groups, even though this is a goal that we may never reach.
- Make communities as convenient as online surveys: be more flexible and tailor communities around member preferences.
- Make communities as fulfilling as having a real person there: consider doing an offline event where the community gather together or provide visits to the factory to get a tour on how the products are made.
Posted by Jeffrey Henning on Tue, Oct 27, 2009
 At the 2009 ESOMAR Online Research conference, James Kennedy from BrainJuicer presented an overview of his vision of the types of community platforms.
"We are witnessing a shift in corporate behavior from where they get ideas for products and services," said Kennedy. At P&G, success rate for new products climbed from 20% in 2000 to 60% in 2008 as P&G embraced consumer-generated ideas, and P&G expects 50% of products to be from consumer-generated ideas in 2010.
Community and crowdsourcing are two new sources of innovation that organizations need to determine how to make work. It's important to have a clear understanding of goals, who you are engaging with and how you are working with them. Kennedy presented the following taxonomy of communities [simplified below]:
|
|
Open |
Closed |
|
Independent |
Independent Open Community |
Multi-Client MROC |
|
Branded |
Branded Open Community |
Branded MROC |
- Branded open communities are a bar; "if you build it, they will come". Some brands have the ability to attract participants to be researched, such as MyStarbucksIdea, which has 75,000 ideas with only 350 ideas implemented. (This represents a lot of new ideas that have been engaged, even though it is only a small percent.)
- Independent open communities are like matchmaking for the B2B world, where people post problems and others post solutions. See NineSigma for an example.
- Branded MROCs are an arranged marriage, where the communities rely on quite close affinities with the brand.
- Multi-client MROCs are what BrainJuicer does with its JuicyBrains innovation community, providing access to 10,000 members across a wide range of categories. The five-phase methodology provides for insight exploration (3 weeks), concept generation (3 weeks), a harvest workshop, a concept clinic (2 weeks) and concept validation. Applications include researching problems, habits, brand associations, test diaries, ideation, concept improvement, claims and positioning, product naming and loyalty ideas.
Using JuicyBrains, Philips Healthcare researched 18 people suffering serious respiratory illness and 5 relatives of such people, asking each to document diagnosis, symptoms and treatment. This study was private and not accessible to other members of the community. Philips Healthcare drew functional insights as well as emotional context, and learnt the importance of better framing how information was presented to patients. Based on the results, Philips is investigating how it can better inform patients about the innovation process.
Posted by Jeffrey Henning on Tue, Oct 27, 2009
 At the ESOMAR Online Research 2009 conference, Volker Bilgram of HYVE AG presented a case study of the Swarovski Enlightened Innovation Research Community, which was designed for the purposes of co-creation, marketing research and community.
- Cocreation. Part of the IRC was a co-creation tool to enable users to configure wristwatch designs from 24 components then selecting and freely placing any of 108 gemstones on the watch. More than one thousand designs were submitted in the first eight weeks, ranging from the simplistic to complex designs that took participants 45 minutes to create. Users could also upload designs created with other software.
- Marketing research. Designs were presented in galleries and rated by three groups: by users, by a jury of Swarovski designers and by a jury of external designers. Users could submit a short evaluation (award 1-5 points) or a detailed evaluation ("vote and win"). Members were invited to comment on one another's designs. Besides stated preferences, data was mined from the configuration tool for frequency of selection of components and gemstones. In addition, preferred combinations were identified.
- Community. The IRC was an open community requiring completion of a short registration form. Users uploaded their own avatars. Banners on design-related websites were used for initial recruitment, which led to buzz to spread interest in the community virally. More than 2000 participants contributed to the contest from 48 countries.
In the post-processing, the winning design was produced as a prototype and presented at a leading trade-show fair in Switzerland. A trend booklet of the MR findings was used for B2B marketing by Swarovski. The community was so successful that Swarovski ran a follow-on community and will continue community research initiatives.
Posted by Jeffrey Henning on Tue, Oct 06, 2009
 Mike Masnick, CEO of Floor64.com, started the blog TechDirt in 1997, which is now rated #75 on Technorati and #53 on BlogPulse. Mike is also the author of Approaching Infinity, about the rise of new abundances: abundant access to ideas and people, with people abundantly willing to join communities, abundantly willing to share, in ways that weren't even possible a few years ago.
Despite the title of his AMA MRC presentation, Mike said, "Don't say that you are building communities, but that you are enabling communities." Thanks to the Internet, communities easily assemble to discuss diverse and oftentimes narrow topics.
Mike contends that what is different for market researchers about online communities is that researchers must engage the audience. Listening and watching in a community is called lurking, which is a bad word in the community world, because it means that you are not participating. Instead, through engagement, you can find out what people really want, not just what they say they want. To truly understand a community, you have to really engage that community. Think of the added level of understanding you have of a friend compared to an acquaintance: engaging a community offers that greater level of detail over simply observing a community.
Engagement builds trust and respect and opens up opportunities for creative dialog.
Posted by Jeffrey Henning on Sun, Oct 04, 2009
Today at the AMA MRC (Marketing Research Conference), I presented a preconference workshop on proprietary panels and online communities. Interestingly, 80% of the attendees were looking at building or expanding proprietary panels and communities of their own customers and prospects.
Here are some of the questions that came up in the session itself and one-on-one with attendees (with links to some answers):
Since it was a three-and-a-half-hour session, I did something new as far as event-evaluation research goes: after the first hour, I gave each attendee a four-question survey, asking what I should change about how I was presenting, what I shouldn't change, how satisfied they were and how enjoyable the session was so far. I took to heart the advice of Jim Davies, principal analyst with Gartner; he had said in his EFM presentation at the Gartner CRM Summit a few weeks ago that too few organizations gather feedback during the experience itself. Based on the survey results, I made the rest of the session more interactive and covered some topics in more detail than I had originally planned.
I loved the additional interaction, which exposed all of us to more points of view. I walked away with six ideas for future blog posts to discuss aspects of panel and online community management that I have neglected so far. I also learnt a widespread fear: that management will look to an online community for all its answers, leaping to conclusions from community research without validating that those insights represent widespread sentiments. We agreed that as market researchers we need to continue to educate our coworkers and management teams about the differences between quantitative and qualitative research, so that organizations make the right decisions. Using panels and online communities as complements to one another is a key way to accomplish this.
Posted by Jeffrey Henning on Mon, Jun 29, 2009
 Far and away, the most common objection we hear from organizations considering extending their survey panel into an online community is that community members will hear each other’s negative feedback. When you survey customers, employees or other key constituencies, only the survey administrator sees all the negative comments. When you move to an online community, anyone who logs in can see the negative comments. This is more an opportunity than an issue. In fact, the absolute value of negative feedback is positive. Here’s why: - Criticisms provide authenticity. Imagine, for a moment, that you hosted an online community and never received any bad reviews or comments. Since every organization and its product and services has shortcomings, an online community where those shortcomings were never discussed would seem unauthentic. It would seem like a sham or a marketing exercise, rather than a true community.
- Negative feedback is scarcer than positive feedback. That scarcity makes it worth more. Online communities excel at generating thousands of positive ideas—more ideas than your organization can implement. Negative feedback is rarer than organizations realize and often clusters around key areas that your organization has ignored or handled poorly; negative feedback gives you a chance to prioritize these issues and focus on improving them.
- Negative feedback is actionable. Negative feedback gives you an opportunity to respond. If a customer tells you ten features they want in the next edition of your product or service, there is nothing you can do for them today but to let them know you are listening and building a list. If they complain about a product damaged in shipping, or a mistake on an invoice, or confusion around a feature, you can immediately help them resolve the issue. Quick resolution of these issues demonstrates that your organization listens and cares.
- Better to manage negative feedback on your turf. The fact of the matter is that people are commenting on your organization all over the World Wide Web: on Twitter, on Facebook, on LinkedIn and on the next big social networking site that we haven’t heard of yet. Brand monitoring solutions can catch some of this commentary, but not all of it: many social networks only show designated friends each other’s comments. Right now on Facebook there’s a long discussion going on about why your organization is awful to do business with, and you will never see or be able to respond to that discussion. When the criticism happens on your site, you can instantly read and respond.
For further reading:
Posted by Jeffrey Henning on Sat, Jun 13, 2009
 Kate Dibben, the Online Communications Officer of Education.au, mentions in passing using “Six Thinking Hats” within communities in her presentation Making Online Communities Work. It’s a good point, worthy of elaboration. If you’ve ever had important conversations in your communities run aground, then you might find the approach useful, especially in smaller private communities. Conversations in online communities, just like conversations around the water cooler, the soccer field or the bar stool, can segue from tangent to tangent to tangent. Oftentimes the journey itself is enjoyable, except when the conversation ends up back where it started, and you’ve made no progress on resolving the issue at hand. Sometimes such conversations are an awkward mix of brainstorming, negativity, new information, misplaced emotion, positive feedback and meta-discussions. This can be especially true in feedback communities, where you’re taught to value the feedback, whatever form it takes. For those times when an issue is being discussed for which you need to put in place an action plan, keep conversations on track by asking participants to take turns wearing the metaphorical Hats. - White Hat – “White is neutral and objective. The white hat is concerned with objective facts and figures.”
- Red Hat – “Red suggests anger (seeing red), rage and emotions. The red hat gives the emotional view.”
- Black Hat – “Black is somber and serious. The black hat is cautious and careful. It points out the weaknesses in an idea.”
- Yellow Hat – “Yellow is sunny and positive. The yellow hat is optimistic and covers hope and positive thinking.”
- Green Hat – “Green is grass, vegetation, and abundant, fertile growth. The green hat indicates creativity and new ideas.”
- Blue Hat – “Blue is cool, and it is also the color of the sky, which is above everything else. The blue hat is concerned with control, the organization of the thinking process, and the use of the other hats.”
A moderator can open a discussion by laying out a sequence through the hats: “We’re going to start by each metaphorically wearing the White Hat and providing information that we have about this feature and how our organization uses it. After that, and only when we are done providing objective information, we will wear the Green Hat and imagine how this feature could be improved. Then we will wear the Yellow Hat and talk about the benefits of the suggestions. Then and only then will we wear the Black Hat and look at the weaknesses of what we’ve talked about. Before we wrap up, we will put on the Red Hat and talk about how the issues we’ve discussed make us feel. Finally, we will wear the Blue Hat and come to a conclusion about what we think our priorities should be.”  As children we were told “to put on our thinking caps”, and De Bono’s Six Thinking Hats provide a simple but useful framework for helping groups think through issues. The Six Thinking Hats can be thought of as traffic cones to help you guide the conversation to its destination. Try it in your online community the next time your discussions need a little structure.
Error sending email
Email sent successfully
|