Posted by Jeffrey Henning on Sun, Feb 07, 2010
For many organizations, panel management is the first step on a path to building online research communities. Once panelists have become accustomed to being surveyed, a natural next step for them is to log into an online portal where they can interact with one another. While not every panelist wants to take the time to participate in an online community, a vocal minority does; their participation will give your organization a rich source of qualitative information.
The great advantage of an online community over a panel is that community members will raise and discuss issues that your organization would have never thought to have researched. This is a wonderful way to develop a broader understanding of your customers. That said, just because an issue becomes
a popular topic in your new online community does not mean that it is an issue across your customer base.
The word quantitative in quantitative research means that you can quantify the wider population based on the research results. The reasons most communities are not representative is because of the self-selection bias for joining the community (similar to the self-selection bias that makes online polls unrepresentative). Even in closed communities, those who agree to participate in communities may represent positive or negative extremes, since agreeing to participate in a community is even more time consuming than agreeing to take a survey. Further, most community managers do not ensure that the community demographics match the overall audience demographics.
For all these reasons, to achieve representativeness, and determine how widespread views of the community are in your wider customer base, you must reach beyond the active community members.
Properly performed panel management can provide a critical source of quantitative research for B2B market researchers and ecommerce sites. In fact, panel management remains of critical importance even after--especially after--an organization has formed online communities.
The best mix of qualitative and quantitative research is to use the online community for qualitative research and to then use the panel for quantitative research.
Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, Customers as Confidants: Customer Panel Management Made Easy.
Posted by Jeffrey Henning on Sun, Jan 31, 2010
In his blog post "Practice Makes Perfect", Reg Baker, COO of Market Strategies, describes the winter Public Opinion Quarterly article, "National Surveys via RDD Telephone Interviewing Versus the Internet: Comparing Sample Representativeness and Response Quality" by Linchiat Chang and Jon A. Krosnick. Reg describes the actual experiment itself [I'd added explanatory links]:
As a broad summary...the online panel results showed less satisficing, less social desirability, better self-reports, and greater internal consistency than did the results from the RDD telephone sample. The online panel folks were just better at doing surveys.
As is typical of a paper by Krosnick, it contains an excellent summary of previous research on research. For our recent research webinar, "Customers as Confidants: Customer Panel Management Made Easy", I distilled this summary into the following high-level pros and cons of panelist participation in surveys.
Pros: Practice Effects
- Regularly answering surveys may enable respondents to improve the accuracy of their responses.
- Panel members may become more introspective and self-aware, improving their self-reporting.
- Respondents' answers to attitudinal questions also improve with practice.
Cons: Panel Conditioning
- The "stimulus hypothesis" that asking about future activity prompts that activity (e.g., asking participants if they will vote actual prompts greater turnout).
- Past surveys taken by panelists make those panelists less like the general population being researched.
- Panelist attrition nonrandomly affects panel representativeness, as members withdraw from panels based on the survey topics they've been subjected to in the past.
Overall, the research seems to indicate that the benefits of the practice effects outweigh the caveats of panel conditioning. Refer to the
Chang and Krosnick article for full details.
Posted by Jeffrey Henning on Sat, Jan 23, 2010
Once your organization puts a basic panel in place, you can begin gathering research more systematically and more regularly. You will see greater response rates as panelists are randomly selected to occasional surveys rather than being invited to every survey.
The results of the Vovici/CGA Customer Experience IQ study show which panel management practices are the most important to focus on. Each best practice was correlated to a customer loyalty index, in order to determine which practices had the highest positive correlation to loyalty.
For panel management, integrating data from CRM into the panel had the highest positive correlation (0.57) to customer loyalty, though it was practiced by comparatively few organizations. By integrating data, surveys appear smarter and more engaging to respondents, who are not asked to provide information that they have provided in the past; surveys with integrated CRM data can branch to relevant sections of the questionnaire based on that hidden data, making the survey more targeted.
Panel management enables organizations to coordinate feedback projects across the entire organization (0.47 correlation to loyalty), reducing survey fatigue by minimizing surveying of the panel. Using the panel to develop a 360 degree view of customer experience is also a key best practice (0.41 correlation), as is using the panel to conduct trend analyses, tracking customer attitudes over time (0.40 correlation).
The results of the CE IQ study will help you prioritize which best practices you should put into place first.
Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, "Customers as Confidants: Customer Panel Management Made Easy".
Posted by Jeffrey Henning on Mon, Jan 11, 2010
Ready to start experiencing the benefits of panel management? Panels can be as easy as 1, 2, 3.
The three essential components of a panel are:
- A central email list for surveys
- An ability to unsubscribe from the panel
- A record of panelists' survey activity
Central Email List for Surveys
Many organizations start with manual panel management for their customer research. They build their panel by importing into a database files with customer profiles. On the plus side, this is quick to set up; on the negative side, this is time-consuming as they try to keep it in sync with the list of customers as new customers are added, old customers move on and existing customers change their contact information.
Automatic panel management typically involves using a file synchronization utility that periodically exports files from a CRM system and then imports new and updated records into the panel. This can be done use FTP transfer of files or by use a SOAP API (a web integration call). Once this is done, the panel is always in sync with the customer data; in fact, many of these processes run as is for years with no intervention or update needed. The only negative is that automatic panel management does require an upfront investment in configuration and custom programming followed up by irregular reconfiguration whenever the CRM file layout changes or new fields are added.
Ability to Unsubscribe from the Panel
The second key requirement of a panel is to maintain a list of people who don't want to be on the list. For U.S. organizations, this suppression list is not just a courtesy but is required by the CAN-SPAM Act for many types of email. Dedicated panel management systems all have unsubscribe processes and suppression lists as a standard feature. The unsubscribe process becomes more complicated, however, when integrated with marketing systems.
Many organizations have EMM (Enterprise Marketing Management) platforms that maintain a global suppression list, where individuals can opt out of all bulk email communication with the organization. Too often, this unsubscribe process is an all-or-nothing approach. If your panel needs to integrate with Marketing's unsubscribe systems, make sure to treat "Surveys" as a discrete channel. Let the reader opt out of some but not all channels of communication: list the available types of mailings and the current email settings and offer a separate channel for "Feedback Surveys." Most individuals are willing to provide feedback, and may unsubscribe from every channel except this one. It's worth asking them.
Record of Panelists' Survey Activity
Third, and finally, a panel management system will track the frequency with which individual panelists have been invited to take surveys and how often each individual has completed surveys. A more sophisticated panel management platform will also include panel health scores to look at the participation rates of panelists individually and in aggregate.
Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, Customers as Confidants: Customer Panel Management Made Easy or sign up for one of our upcoming panel research webinars.
Posted by Jeffrey Henning on Wed, Nov 18, 2009
A panel is a group of people with relevant backgrounds who agree to participate in surveys. Businesses can organize a panel for each group of key stakeholders: customers, employees, resellers, partners, prospects, etc.
Because panelists agree in advance to participate in surveys and feedback efforts, they become almost a guaranteed source of information for the sponsoring organization. Customers typically participate in the panel because they value their relationship with the sponsor, and they appreciate the additional information, influence and early access that comes from participating in the panel. For general market panels, panelists instead often look to merchandise and cash rewards, though even for many general panels people participate because they want to make their views heard.
How does a panel work? You invite an individual to participate in a panel and communicate the ground rules:
- What they will be surveyed about
- How you'll use the information
- How frequently they'll be asked to participate in surveys
- Why it's important that they participate
- How they can opt-out if they change their mind
- What's in it for them
Once you gain their permission, you invite the panelist to complete a registration survey, which will gather detailed demographic or firmographic information. This information can be used to target individual surveys to panel subsegments and also provides for rich opportunities in cross-tabulating survey results.
In contrast to online communities, in a panel, members communicate only with the sponsoring researcher, through the medium of the surveys they are sent. In online communities, members can engage in discussions with one another through an online portal.
Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, Customers as Confidants: Customer Panel Management Made Easy.
Experience Vovici Request a Call from a Vovici Rep Request a Demo with a Vovici Rep
Posted by Jeffrey Henning on Wed, Oct 28, 2009
Joel Rubinson, the Chief Research Officer of the ARF, discussed ARF quality initiatives. The ARF ORQC (Online Research Quality Council) was assembled to address the issue of panel quality. The committees developed "Foundations of Quality", a research-on-research project in October and November 2008 conducting 100,000 surveys across 17 U.S. panels, supplemented by phone and mail research.
The research revealed that all seventeen panels could retest their results reliably, but results varied significantly from panel to panel. Switching panels or using a panel undergoing major changes could change the results. The purchase intent for a concept correlates to panelist longevity (panels vary significantly by longevity), but no weighting scheme removed this variation when doing multi-panel sourcing. Rebutting assumptions, taking multiple surveys per month (3 to 10) was actually good for respondent engagement, putting the professional in the term professional respondent, which is usually a disparaging terms. Most panelists were not in it for the money, but those who were provided lower quality answers.
An industry-solutions committee started translating the insights from this research into an action plan, the QeP. The QeP (Quality Enhancement Process) v 1.0 offers templates, definitions, metrics and declarations to bring structure to conversations between buyers and sellers about online panel quality. The QeP is designed as a process to help buyers and sellers meet the shared goal of providing valid and consistent data. QeP is designed to encourage innovation that improves data validity across the online research ecosystem, which includes marketing and research departments on the client side, management consultants, suppliers and subcontractors to those suppliers.
The QeP looks at standards of quality at the panel level, sample/study level and the survey/response-quality level.
- At the panel level, panel providers should document recruitment methods, incentive systems, panelist profile, privacy policy, survey QA standards, etc. When panels merge, buyers should be wary of doing trend analysis with results from a predecessor panel.
- At the sample/study level, document a consistent sampling plan, promote sample consistency and eliminate duplicate survey taking in any form (deduping households, not just respondents).
- At the survey/response-quality level, control for panelist longevity.
The next step for ARF is to pilot test the QeF to ensure that the templates are clear, the documents are workable and the reports are useful and comprehensive. These results will drive future evolution of the QeP.
Update - Joel Rubinson uploaded his presentation to SlideSlide:
Posted by Jeffrey Henning on Sun, Oct 04, 2009
Today at the AMA MRC (Marketing Research Conference), I presented a preconference workshop on proprietary panels and online communities. Interestingly, 80% of the attendees were looking at building or expanding proprietary panels and communities of their own customers and prospects.
Here are some of the questions that came up in the session itself and one-on-one with attendees (with links to some answers):
Since it was a three-and-a-half-hour session, I did something new as far as event-evaluation research goes: after the first hour, I gave each attendee a four-question survey, asking what I should change about how I was presenting, what I shouldn't change, how satisfied they were and how enjoyable the session was so far. I took to heart the advice of Jim Davies, principal analyst with Gartner; he had said in his EFM presentation at the Gartner CRM Summit a few weeks ago that too few organizations gather feedback during the experience itself. Based on the survey results, I made the rest of the session more interactive and covered some topics in more detail than I had originally planned.
I loved the additional interaction, which exposed all of us to more points of view. I walked away with six ideas for future blog posts to discuss aspects of panel and online community management that I have neglected so far. I also learnt a widespread fear: that management will look to an online community for all its answers, leaping to conclusions from community research without validating that those insights represent widespread sentiments. We agreed that as market researchers we need to continue to educate our coworkers and management teams about the differences between quantitative and qualitative research, so that organizations make the right decisions. Using panels and online communities as complements to one another is a key way to accomplish this.
Posted by Jeffrey Henning on Fri, Sep 04, 2009
I’ve ignored many of the initiatives to improve the quality of third-party online panels. To me, these initiatives are laughable. Yes, you should…
- Seek to identify panelists participating in the same survey multiple times under different names
- Remove respondents who speed through their answers
- Have a broad-based demographic representation so that you do not need to weight individual respondents
But these simply put lipstick on the piggy bank. They make it easier for organizations to continue to put cost before quality and to justify doing research on the cheap with third-party panels. “See? The panel companies are working hard to ensure consistent high quality!”
Um, a consistent high quality convenience panel is certainly better than a low quality convenience panel. But it’s still a pig. Er, piggy bank: a cheap alternative to a
random sample.
The laws of mathematics have not been repealed: a convenience sample cannot be used to extrapolate to any target audience. A convenience sample is representative of its respondents only. This point keeps getting lost, as I saw last year at the MRA Conference at the presentation
What's the Catch? Does Sample Sourcing Matter:
A pointed question from the audience said that probability sampling was the theoretical basis for the projectability of survey research and asked what the scientific underpinnings were for assuming that Internet research was similarly representative. Melanie [the presenter] answered that replicability is emerging as the standard instead of randomization and that the results from her research were replicable.
What "irrational exuberance" was to NASDAQ, the third-party online panel is to MR.
This week, Gary Langer, director of polling at ABC News, writes
in his column:
A new study led by Stanford University researchers raises doubts about the accuracy of one of the most common forms of survey research, polls done among people who sign up to fill in questionnaires via the internet in exchange for cash and gifts.
In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two surveys based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.
While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.
Most Vovici customers are surveying house lists of customers, employees, resellers and other key constituencies. It’s very easy to do a random survey of employees when you have the email address of every employee and have empaneled the list of employees by synchronizing your HRIS. For surveys of prospects, many organizations are using the web for all lead generation and can easily field random samples of prospects. Unless you’re an e-commerce or SaaS business, though, it is more difficult to build a representative house list of customers that you can then random sample: check out these tips for creating and managing
representative email lists of your customer base.
Putting in regular processes to build a quality house list is like setting up automatic monthly withdrawals from checking to savings: better than the panel piggy bank as way to save research costs in the long run. Building such a house list is a sound investment towards conducting quality, representative survey research.
Posted by Jeffrey Henning on Wed, May 13, 2009
Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.
Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don't, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.
Organizations with unrepresentative email lists need to take the following steps for web-survey success:
- Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
- Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
- Conduct a telephone study or paper survey of customers for which your organization doesn't have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
- Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication.
- Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.
What steps have you taken to make sure that your web surveys are representative of your customer base?
Posted by Jeffrey Henning on Wed, Apr 22, 2009