Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
|
 RSS Feed
Posted by Jeffrey Henning on Wed, Nov 18, 2009
 A panel is a group of people with relevant backgrounds who agree to participate in surveys. Businesses can organize a panel for each group of key stakeholders: customers, employees, resellers, partners, prospects, etc.
Because panelists agree in advance to participate in surveys and feedback efforts, they become almost a guaranteed source of information for the sponsoring organization. Customers typically participate in the panel because they value their relationship with the sponsor, and they appreciate the additional information, influence and early access that comes from participating in the panel. For general market panels, panelists instead often look to merchandise and cash rewards, though even for many general panels people participate because they want to make their views heard.
How does a panel work? You invite an individual to participate in a panel and communicate the ground rules:
- What they will be surveyed about
- How you'll use the information
- How frequently they'll be asked to participate in surveys
- Why it's important that they participate
- How they can opt-out if they change their mind
- What's in it for them
Once you gain their permission, you invite the panelist to complete a registration survey, which will gather detailed demographic or firmographic information. This information can be used to target individual surveys to panel subsegments and also provides for rich opportunities in cross-tabulating survey results.
In contrast to online communities, in a panel, members communicate only with the sponsoring researcher, through the medium of the surveys they are sent. In online communities, members can engage in discussions with one another through an online portal.
Want to learn how to set up your own customer panel? Download a complimentary copy of my white paper, Customers as Confidants: Customer Panel Management Made Easy. Vovici Survey Software and Web Survey: Request a Call l Request a Demo
Posted by Jeffrey Henning on Wed, Oct 28, 2009
 Joel Rubinson, the Chief Research Officer of the ARF, discussed ARF quality initiatives. The ARF ORQC (Online Research Quality Council) was assembled to address the issue of panel quality. The committees developed "Foundations of Quality", a research-on-research project in October and November 2008 conducting 100,000 surveys across 17 U.S. panels, supplemented by phone and mail research.
The research revealed that all seventeen panels could retest their results reliably, but results varied significantly from panel to panel. Switching panels or using a panel undergoing major changes could change the results. The purchase intent for a concept correlates to panelist longevity (panels vary significantly by longevity), but no weighting scheme removed this variation when doing multi-panel sourcing. Rebutting assumptions, taking multiple surveys per month (3 to 10) was actually good for respondent engagement, putting the professional in the term professional respondent, which is usually a disparaging terms. Most panelists were not in it for the money, but those who were provided lower quality answers.
An industry-solutions committee started translating the insights from this research into an action plan, the QeP. The QeP (Quality Enhancement Process) v 1.0 offers templates, definitions, metrics and declarations to bring structure to conversations between buyers and sellers about online panel quality. The QeP is designed as a process to help buyers and sellers meet the shared goal of providing valid and consistent data. QeP is designed to encourage innovation that improves data validity across the online research ecosystem, which includes marketing and research departments on the client side, management consultants, suppliers and subcontractors to those suppliers.
The QeP looks at standards of quality at the panel level, sample/study level and the survey/response-quality level.
- At the panel level, panel providers should document recruitment methods, incentive systems, panelist profile, privacy policy, survey QA standards, etc. When panels merge, buyers should be wary of doing trend analysis with results from a predecessor panel.
- At the sample/study level, document a consistent sampling plan, promote sample consistency and eliminate duplicate survey taking in any form (deduping households, not just respondents).
- At the survey/response-quality level, control for panelist longevity.
The next step for ARF is to pilot test the QeF to ensure that the templates are clear, the documents are workable and the reports are useful and comprehensive. These results will drive future evolution of the QeP.
Update - Joel Rubinson uploaded his presentation to SlideSlide:
Posted by Jeffrey Henning on Sun, Oct 04, 2009
 Today at the AMA MRC (Marketing Research Conference), I presented a preconference workshop on proprietary panels and online communities. Interestingly, 80% of the attendees were looking at building or expanding proprietary panels and communities of their own customers and prospects.
Here are some of the questions that came up in the session itself and one-on-one with attendees (with links to some answers):
Since it was a three-and-a-half-hour session, I did something new as far as event-evaluation research goes: after the first hour, I gave each attendee a four-question survey, asking what I should change about how I was presenting, what I shouldn't change, how satisfied they were and how enjoyable the session was so far. I took to heart the advice of Jim Davies, principal analyst with Gartner; he had said in his EFM presentation at the Gartner CRM Summit a few weeks ago that too few organizations gather feedback during the experience itself. Based on the survey results, I made the rest of the session more interactive and covered some topics in more detail than I had originally planned.
I loved the additional interaction, which exposed all of us to more points of view. I walked away with six ideas for future blog posts to discuss aspects of panel and online community management that I have neglected so far. I also learnt a widespread fear: that management will look to an online community for all its answers, leaping to conclusions from community research without validating that those insights represent widespread sentiments. We agreed that as market researchers we need to continue to educate our coworkers and management teams about the differences between quantitative and qualitative research, so that organizations make the right decisions. Using panels and online communities as complements to one another is a key way to accomplish this.
Posted by Jeffrey Henning on Fri, Sep 04, 2009
I’ve ignored many of the initiatives to improve the quality of third-party online panels. To me, these initiatives are laughable. Yes, you should…
- Seek to identify panelists participating in the same survey multiple times under different names
- Remove respondents who speed through their answers
- Have a broad-based demographic representation so that you do not need to weight individual respondents
But these simply put lipstick on the piggy bank. They make it easier for organizations to continue to put cost before quality and to justify doing research on the cheap with third-party panels. “See? The panel companies are working hard to ensure consistent high quality!”
Um, a consistent high quality convenience panel is certainly better than a low quality convenience panel. But it’s still a pig. Er, piggy bank: a cheap alternative to a random sample.
The laws of mathematics have not been repealed: a convenience sample cannot be used to extrapolate to any target audience. A convenience sample is representative of its respondents only. This point keeps getting lost, as I saw last year at the MRA Conference at the presentation What's the Catch? Does Sample Sourcing Matter:
A pointed question from the audience said that probability sampling was the theoretical basis for the projectability of survey research and asked what the scientific underpinnings were for assuming that Internet research was similarly representative. Melanie [the presenter] answered that replicability is emerging as the standard instead of randomization and that the results from her research were replicable.
What "irrational exuberance" was to NASDAQ, the third-party online panel is to MR.
This week, Gary Langer, director of polling at ABC News, writes in his column:
A new study led by Stanford University researchers raises doubts about the accuracy of one of the most common forms of survey research, polls done among people who sign up to fill in questionnaires via the internet in exchange for cash and gifts.
In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two surveys based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.
While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.
Most Vovici customers are surveying house lists of customers, employees, resellers and other key constituencies. It’s very easy to do a random survey of employees when you have the email address of every employee and have empaneled the list of employees by synchronizing your HRIS. For surveys of prospects, many organizations are using the web for all lead generation and can easily field random samples of prospects. Unless you’re an e-commerce or SaaS business, though, it is more difficult to build a representative house list of customers that you can then random sample: check out these tips for creating and managing representative email lists of your customer base.
Putting in regular processes to build a quality house list is like setting up automatic monthly withdrawals from checking to savings: better than the panel piggy bank as way to save research costs in the long run. Building such a house list is a sound investment towards conducting quality, representative survey research.
Posted by Jeffrey Henning on Wed, May 13, 2009
 Some organizations, such as ecommerce sites like Amazon and EBay, are fortunate in that they have the email addresses of every single one of their customers. When they invite a random selection of customers to participate in a web survey, they can be assured that the results are truly representative of their customer base.
Most organizations are not so fortunate. A traditional B2B vendor might have the email addresses for perhaps half of its customers. A consumer brand might have email addresses for fewer than 10% of its customers. For these organizations, web surveys with random samples are only representative of the customers for whom they have email addresses. Such organizations must be very careful when presenting their results to not generalize them to all customers. Significant differences probably exist between the group of customers for which they have email addresses and the group for which they don't, especially since email addresses are typically collected as part of "loyalty" (frequent-buyer) programs. As a result, surveys of such email lists will overstate satisfaction, repurchase likelihood and willingness to recommend.
Organizations with unrepresentative email lists need to take the following steps for web-survey success: - Describe the survey not as representative of all customers but of the target population for which the organization has email addresses.
- Work with the marketing department to increase the percent of customers and prospects who have provided email addresses.
- Conduct a telephone study or paper survey of customers for which your organization doesn't have email addresses to determine their demographics, firmographics and attitudes. This can be used to contrast this group with the group accessible by email.
- Conduct more-frequent surveys of members of the customer loyalty program. Such programs typically use email for most communication.
- Don't put faith in polls placed on your corporate web site; such polls are unrepresentative.
What steps have you taken to make sure that your web surveys are representative of your customer base?
Posted by Jeffrey Henning on Wed, Apr 22, 2009
Posted by Jeffrey Henning on Thu, Nov 13, 2008
 Esteban Kolsky, a principal analyst with Evergence, and one of the patron saints of EFM, asked, in the comment section to my post Panel Management 101, "How does the integration work just for panel management? I have integration as critical for EFM - but not directly related to panel management... am I missing something?"
Most panel management software systems have profiles of panelists, where demographic, transactional and attitudinal information is stored for each panelist. Integrating other data sources with these profiles provides users of survey software a number of key benefits. Data integration lets you shorten questionnaires by piping in answers to profile questions behind the scenes. It lets you pipe in questions that you can use for more detailed cross-tabulation and survey analysis. It also provides you with questions that you can use for "on demand segmentation", targeting specific groups for special surveys. And data integration saves your panelists from having to re-enter information your organization already knows about them.
 Here are some examples of the type of panel-management integration work that we've done.
- CRM Data - Using CRM connectors to synchronize a demographic profile of an empanelled customer with information from their customer-relationship management records.
- Transactional Data - Storing the most recent customer transaction or series of transactions in the profile. For help-desk surveys, storing the last time the customer contacted technical support, along with details of the type of contact, the resolution, the date of the contact, and so forth. For course-evaluation surveys, storing the most recent classes attended by the student. For retail-satisfaction surveys, storing details about the most recent purchase.
- Benchmark Data - Importing records from past surveys for use as historical benchmarks (especially from phone or paper surveys or other methodologies that predated web surveys).
While enterprise feedback management systems can certainly have other types of integration (we've integrated with third-party communities, besides our online community software; we've exported survey data back to CRM systems), most data-integration opportunities have actually been around panel management profiles. I hope that answers your question, Esteban!
And if you find my blog useful, I think you will find Esteban's blog worth reading as well.
Posted by Jeffrey Henning on Wed, Nov 05, 2008
 Why should you consider adding panel management to your survey software?
- Your response rate to web surveys (the percent of people who complete the survey) is declining.
- Your respondents, like those of this auto manufacturer, are complaining that they are being sent too many surveys.
- You need to easily target surveys to segments and subsegments of your customers for detailed survey analysis.
- Your respondents are complaining that surveys are too long. You have to ask respondents questions you already know the answer to, such as the products of yours that they use and the services that they subscribe to (for consumers: their gender, age, location; for businesses: their size, industry, region).
- Your survey authors have to wait on IT to export mailing lists from your other systems.
Panel management assists with each of these items:
- By empanelling all potential respondents in a central database, you eliminate the need to rely on IT resources to generate recipient lists.
- By centralizing "touch management" for your survey research, you can make certain not to oversurvey panelists. Oversurveying is a leading cause of declining response rates.
- By profiling potential respondents in detail, you can segment "on demand" when sending out survey invitations.
- By integrating respondent profiles with CRM systems through CRM connectors, you can embed that customer data as hidden fields within a survey, streamlining the survey from the respondent's perspective.
Once usage of surveys has spread across an organization, panel management is a logical next step to begin to bring that research under control, while moving along the path to implementing online communities and true enterprise feedback management.
Posted by Jeffrey Henning on Tue, Sep 23, 2008
 Yesterday I described the "Twists and Turns on 21st Century Internet Express" presentation (The Survey Taker is King) at the AMA 2008 Marketing Research Conference. Many of the problems that face traditional market-research surveys go away when a corporation is able to survey its customers, employees or resellers, rather than a third-party population. Enterprise-feedback management systems and feedback communities alleviate many of these concerns.
The audience was polled for their concerns with the question, "What should be done to increase online survey cooperation rates?"
- Make surveys more engaging (50%) - Actually, the most engaging surveys are those for relationships that the respondent has the greatest involvement in. The challenge for MR is getting people they have no relationship with to take a survey. Employee surveys typically have the highest participation rates, because employees invest a lot in their work; customer surveys have solid response rates, with greater response rates the more money is spent; surveys of prospects or general consumers have the lowest response rates. When you're surveying your community, you start with engaged survey takers; your job is to respect that engagement and not dilute it through bad surveys.
- Limit the number of questions/time to complete (26%) - Traditional market research surveys tend to be long: researchers know nothing about the respondent and need to ask them detailed demographic and firmographic questions; researchers will often ask a question multiple ways to develop a more reliable index that can be used for tracking; researchers will have many grid questions, gathering different aspects of attitude for a common list of topics. Many of these reasons for length go away for the community researcher: demographic and other profile information can be piped into the survey, rather than prompted for; individual key questions are asked rather than index-question groups; finally, researchers can survey often, rather than getting every answer from this single survey.
- Better target surveys to individuals (15%) - Using the profile data in a community or enterprise feedback management system, community researchers can invite the right respondents, where market researchers need to invite everyone, then ask questions to screen out the wrong respondents.
- Increase the use of incentives (7%) - Incentives are less necessary in a feedback community, where respondents are more motivated by their engagement than by extrinsic rewards.
The reverse of the question was also asked to session attendees: "What factors are contributing to lower online co-operation?"
 After "Length of time required to complete surveys" at 51%, "Frequency of invitations issued to survey participants" (30%) and "Lack of online panel development and active management" (11%) and were the two most popular choices. EFM systems are all about touch management, making sure community members and panelists are not oversurveyed. Internet panels, used for generic market research, have forsaken the traditional limits on survey participation, and are further hobbled by the need to invite people to take a screener survey before qualifying for the full questionnaire.
Clearly, for researchers who need to survey their organization's key constituencies rather than the market at large, enterprise feedback management systems and feedback communities eliminate many of the pressing challenges faced by market researchers today.
Traditional researchers haven't needed to be responsible to the respondent, who was a replaceable commodity; for community researchers, respondents are valued customers, employees and partners, and therefore should be treated like royalty.
Posted by Jeffrey Henning on Mon, Sep 01, 2008
As we in the United States celebrate Labor Day, I will rise to the defense of workers everywhere: don't oversurvey them!
About 45% of our customers survey their employees, in a myriad of different ways:
- Annual in-depth employee-satisfaction measurement
- Quarterly snapshot employee-satisfaction measurement
- Benefits satisfaction and prioritization
- Awareness of corporate initiatives
- Sarbanes-Oxley compliance
- Corporate-policy compliance
- Whistle-blower reporting
- Ethics assessments
- Event planning
- Event registration
- Product/service feedback
- Competitive assessments
- IT help-desk satisfaction
- And many more applications
Once an organization adopts a method of surveying its employees, it quickly comes up with ideas for surveys that it never would have considered before: surveys for preferences for the summer picnic, ideas for names for the company newsletter, even lunch orders! All too often, most of these surveys are done of the entire company, because it so easy to do so and it doesn't have any apparent cost.
But it does have an implicit cost: it burns out employees on providing feedback. Response rates begin to drop, and the feedback is more cursory, with fewer in-depth verbatim responses.
The cure is simple but painful at first. Just like Labor Day, give employees a rest. Use an employee community to set up profiles of employees with facts like: the department they work for, their level in the organization, and other attributes you can use to target survey invitations. Then instead of doing censuses - surveys of the entire employee base - either do censuses of specific departments (e.g., invite all sales staff and sales-engineers to complete the assessment of competitors) or do random samples (e.g., for that quarterly snapshot or "take a pulse" survey, use a random sample across the entire organization). This will help preserve the quality of feedback gathered, without belaboring your workforce!
Error sending email
Email sent successfully
|