Posted by Jeffrey Henning on Wed, Apr 22, 2009
Posted by Jeffrey Henning on Thu, Nov 13, 2008
Esteban Kolsky, a principal analyst with Evergence, and one of the patron saints of EFM, asked, in the comment section to my post Panel Management 101, "How does the integration work just for panel management? I have integration as critical for EFM - but not directly related to panel management... am I missing something?"
Most panel management software systems have profiles of panelists, where demographic, transactional and attitudinal information is stored for each panelist. Integrating other data sources with these profiles provides users of survey software a number of key benefits. Data integration lets you shorten questionnaires by piping in answers to profile questions behind the scenes. It lets you pipe in questions that you can use for more detailed cross-tabulation and survey analysis. It also provides you with questions that you can use for "on demand segmentation", targeting specific groups for special surveys. And data integration saves your panelists from having to re-enter information your organization already knows about them.
Here are some examples of the type of panel-management integration work that we've done.
- CRM Data - Using CRM connectors to synchronize a demographic profile of an empanelled customer with information from their customer-relationship management records.
- Transactional Data - Storing the most recent customer transaction or series of transactions in the profile. For help-desk surveys, storing the last time the customer contacted technical support, along with details of the type of contact, the resolution, the date of the contact, and so forth. For course-evaluation surveys, storing the most recent classes attended by the student. For retail-satisfaction surveys, storing details about the most recent purchase.
- Benchmark Data - Importing records from past surveys for use as historical benchmarks (especially from phone or paper surveys or other methodologies that predated web surveys).
While enterprise feedback management systems can certainly have other types of integration (we've integrated with third-party communities, besides our online community software; we've exported survey data back to CRM systems), most data-integration opportunities have actually been around panel management profiles. I hope that answers your question, Esteban!
And if you find my blog useful, I think you will find Esteban's blog worth reading as well.
Posted by Jeffrey Henning on Wed, Nov 05, 2008
Why should you consider adding panel management to your survey software?
- Your response rate to web surveys (the percent of people who complete the survey) is declining.
- Your respondents, like those of this auto manufacturer, are complaining that they are being sent too many surveys.
- You need to easily target surveys to segments and subsegments of your customers for detailed survey analysis.
- Your respondents are complaining that surveys are too long. You have to ask respondents questions you already know the answer to, such as the products of yours that they use and the services that they subscribe to (for consumers: their gender, age, location; for businesses: their size, industry, region).
- Your survey authors have to wait on IT to export mailing lists from your other systems.
Panel management assists with each of these items:
- By empanelling all potential respondents in a central database, you eliminate the need to rely on IT resources to generate recipient lists.
- By centralizing "touch management" for your survey research, you can make certain not to oversurvey panelists. Oversurveying is a leading cause of declining response rates.
- By profiling potential respondents in detail, you can segment "on demand" when sending out survey invitations.
- By integrating respondent profiles with CRM systems through CRM connectors, you can embed that customer data as hidden fields within a survey, streamlining the survey from the respondent's perspective.
Once usage of surveys has spread across an organization, panel management is a logical next step to begin to bring that research under control, while moving along the path to implementing online communities and true enterprise feedback management.
Posted by Jeffrey Henning on Tue, Sep 23, 2008
Yesterday I described the "Twists and Turns on 21st Century Internet Express" presentation (The Survey Taker is King) at the AMA 2008 Marketing Research Conference. Many of the problems that face traditional market-research surveys go away when a corporation is able to survey its customers, employees or resellers, rather than a third-party population. Enterprise-feedback management systems and feedback communities alleviate many of these concerns.
The audience was polled for their concerns with the question, "What should be done to increase online survey cooperation rates?"
- Make surveys more engaging (50%) - Actually, the most engaging surveys are those for relationships that the respondent has the greatest involvement in. The challenge for MR is getting people they have no relationship with to take a survey. Employee surveys typically have the highest participation rates, because employees invest a lot in their work; customer surveys have solid response rates, with greater response rates the more money is spent; surveys of prospects or general consumers have the lowest response rates. When you're surveying your community, you start with engaged survey takers; your job is to respect that engagement and not dilute it through bad surveys.
- Limit the number of questions/time to complete (26%) - Traditional market research surveys tend to be long: researchers know nothing about the respondent and need to ask them detailed demographic and firmographic questions; researchers will often ask a question multiple ways to develop a more reliable index that can be used for tracking; researchers will have many grid questions, gathering different aspects of attitude for a common list of topics. Many of these reasons for length go away for the community researcher: demographic and other profile information can be piped into the survey, rather than prompted for; individual key questions are asked rather than index-question groups; finally, researchers can survey often, rather than getting every answer from this single survey.
- Better target surveys to individuals (15%) - Using the profile data in a community or enterprise feedback management system, community researchers can invite the right respondents, where market researchers need to invite everyone, then ask questions to screen out the wrong respondents.
- Increase the use of incentives (7%) - Incentives are less necessary in a feedback community, where respondents are more motivated by their engagement than by extrinsic rewards.
The reverse of the question was also asked to session attendees: "What factors are contributing to lower online co-operation?"
After "Length of time required to complete surveys" at 51%, "Frequency of invitations issued to survey participants" (30%) and "Lack of online panel development and active management" (11%) and were the two most popular choices. EFM systems are all about touch management, making sure community members and panelists are not oversurveyed. Internet panels, used for generic market research, have forsaken the traditional limits on survey participation, and are further hobbled by the need to invite people to take a screener survey before qualifying for the full questionnaire.
Clearly, for researchers who need to survey their organization's key constituencies rather than the market at large, enterprise feedback management systems and feedback communities eliminate many of the pressing challenges faced by market researchers today.
Traditional researchers haven't needed to be responsible to the respondent, who was a replaceable commodity; for community researchers, respondents are valued customers, employees and partners, and therefore should be treated like royalty.
Posted by Jeffrey Henning on Mon, Sep 01, 2008
As we in the United States celebrate Labor Day, I will rise to the defense of workers everywhere: don't oversurvey them!
About 45% of our customers survey their employees, in a myriad of different ways:
- Annual in-depth employee-satisfaction measurement
- Quarterly snapshot employee-satisfaction measurement
- Benefits satisfaction and prioritization
- Awareness of corporate initiatives
- Sarbanes-Oxley compliance
- Corporate-policy compliance
- Whistle-blower reporting
- Ethics assessments
- Event planning
- Event registration
- Product/service feedback
- Competitive assessments
- IT help-desk satisfaction
- And many more applications
Once an organization adopts a method of surveying its employees, it quickly comes up with ideas for surveys that it never would have considered before: surveys for preferences for the summer picnic, ideas for names for the company newsletter, even lunch orders! All too often, most of these surveys are done of the entire company, because it so easy to do so and it doesn't have any apparent cost.
But it does have an implicit cost: it burns out employees on providing feedback. Response rates begin to drop, and the feedback is more cursory, with fewer in-depth verbatim responses.
The cure is simple but painful at first. Just like Labor Day, give employees a rest. Use an employee community to set up profiles of employees with facts like: the department they work for, their level in the organization, and other attributes you can use to target survey invitations. Then instead of doing censuses - surveys of the entire employee base - either do censuses of specific departments (e.g., invite all sales staff and sales-engineers to complete the assessment of competitors) or do random samples (e.g., for that quarterly snapshot or "take a pulse" survey, use a random sample across the entire organization). This will help preserve the quality of feedback gathered, without belaboring your workforce!
Posted by Jeffrey Henning on Thu, Jun 12, 2008
At the MRA Annual Conference last week, Melanie Courtright, a vice president with DMS Research (an AOL subsidiary), presented the results of a research study she designed contrasting the results gathered from telephone surveys, Internet-panel surveys and Internet-river surveys.
So-called "river surveys" dip into the stream of users visiting thousands of web sites to invite them to take a survey, rather than emailing a pre-built panel of respondents. DMS's Opinion Place river runs promotions across thousands of sites, invites visitors to take a survey, then asks them an 11-question qualifier which helps route each visitor to a specific survey by their demographics. Her title, "What's the Catch?" alludes to catching these respondents from the river. (As she described the Opinion Place river, I had to laugh as I envisioned a catch-and-release program for respondents, where they were released back to the Internet after completing their survey!)
In Phase I of the research, conducted in December and collecting 2,412 responses, phone, panel and river surveys were contrasted. The results uncovered minor attitudinal differences but uncovered significant differences in technology usage and adoption, as would be expected. For instance, 69% of telephone respondents had been using the Internet for at least 5 years, compared to 80% of the river respondents; 22% of phone respondents don't have home Internet access, compared to 1% of the river respondents; 21% of phone respondents don't have a cell phone, compared to 9% of the river respondents.
An additional key area of difference was in the amount of survey participation. Only 4% of phone respondents take surveys weekly, compared to 17% of river respondents and 70% of panel respondents. The average number of surveys per month was 0.3 surveys per phone respondent, 2.9 surveys per river respondent and 16.6 surveys per panel respondent. Clearly, for researchers worried about the affect of professional respondents on results, telephone or river surveys are better choices than third-party Internet panels.
A Phase II study of 3,647 responses was conducted in April across seven panels. It found surprisingly few differences between results derived from specific panels, with the difference again being technology usage, as some panels seem to have more early adopters than others.
For data quality, 21% of panel respondents completed the questionnaire in 7 minutes or less, compared to 5% of river respondents. DMS found quality problems such as answering a trick question wrong or not answering an open-ended question thoughtfully were far more typical of speeders. Melanie said, "If there is one step you take to improve the quality of your responses, take a look at the responses from speeders."
When mapping the survey results to standard benchmarks such as percent of the U.S. population that are married, are students, are Sunday newspaper readers, etc., each survey methodology produced results close to the benchmark. However, each methodology produced very different results on technology usage, again as would be expected given that the methodologies differed by technology.
A pointed question from the audience said that probability sampling was the theoretical basis for the projectability of survey research and asked what the scientific underpinnings were for assuming that Internet research was similarly representative. Melanie answered that replicability is emerging as the standard instead of randomization and that the results from her research were replicable.
Answering the question posed by her presentation title, Melanie echoes Brad Bortner at Forrester to say that sample source doesn't matter as much as people may think. Melanie's recommendation was that third-party panels were best for surveying low-incidence populations, phone surveys were best when concerned about the skew from online methodologies, and river surveys were best for reaching less surveyed Internet populations.
Posted by Jeffrey Henning on Sat, May 31, 2008
Brad Bortner, a principal analyst with Forrester, has written an excellent white paper about the representativeness of online research. Here's an abstract of Does Declining Research Projectability Matter? (emphasis added):
"Conventional wisdom holds that online panels are inherently problematic in supplying projectable findings to the general population - online and offline. Critics assert that online panels aren't representative and that new types of dirt in the data - such as professional survey takers - erode the accuracy of projections. However, it's hard to find any substantive marketing decisions that have gone awry due to using online panel-based research. Why? Because good online panels actually deliver as much accuracy as they give up. Market research professionals should leverage what online panels are good at - speed, cost, and new analytic approaches - and at the same time hedge risk by mixing data collection modes, expanding their analysis framework to create 'what if' models, and understanding that sometimes speed of result trumps accuracy and cost."
Brad writes about third-party panels, which he sees as sufficient for research but suffering from three key problems:
- Wide variances in projectability from different panels
- Underrepresentation of key groups
- Bias introduced by "professional" survey takers
Because proprietary panels are derived from company databases, proprietary panels in contrast are highly representative of the target population of a company's clients and employees. See this blog post last month for a more detailed comparison of the two types of panels: Panelist Quality of External Lists vs. Feedback-Focused Communities.
If you have any concerns about the projectability of online research, I strongly encourage you to buy Brad's report.
Posted by Jeffrey Henning on Fri, Apr 25, 2008
Market researchers are becoming increasingly concerned about online panelists who are participating in surveys for personal gain rather than simply to provide feedback. As a result, vendors are offering new solutions. This month alone Peanut Labs won an award for its Optimus data quality technology and MarketTools introduced its TrueSample certification program.
The MarketTools announcement quotes Doug Doyle, director of market research at Microsoft, as saying, "Microsoft has found that trust in external sample quality [emphasis added] is the number-one challenge facing online market research."
This is the huge difference between surveying external lists and internal lists of customers, resellers, employees and other key constituencies. MarketTools states that "TrueSample is a three-part process aimed at ensuring authenticity in survey respondents. It provides objective assurance that survey respondents are real, unique and engaged."
Feedback-focused communities such as those used by our customers already ensure that survey respondents are "real, unique and engaged."
Real: Because feedback-community members are recruited from existing company records of customers, prospects, employees and investors, you can be assured that these are real people with real relationships with your organization.
Unique: Your organization already expends resources to validate that each customer record in your CRM system is unique, and most likely has similar data-scrubbing procedures already in place for other databases of important constituencies. EFM Community can be synchronized with these databases to leverage your existing investment in ensuring unique responses.
Engaged: The fundamental challenge of external sample research is weeding out respondents who are merely answering questions randomly in order to qualify for prizes. Incentives offered to customers and employees are typically tokens of appreciation that are not going to encourage respondents to distort their answers to qualify; these respondents value their relationships with your organization and want to see those relationships improve. The desire to strengthen that relationship provides the ultimate in engagement!
Such concerns about external list quality provide compelling reasons to develop internal communities for many research initiatives.
Posted by Dean Wiltse on Fri, Nov 16, 2007
I attended the CASRO (Council Survey Research Organizations) conference in Scottsdale during October. This was my 5th conference in the last seven years; my first conference was in 2001. It was held in Amelia Island during early October, a few weeks after 9/11. I drove from New York to Florida because I just wasn't ready to fly. I had flown flight 91 out of Boston many times and I had been in NYC the day before the attack. I also knew someone that lost her husband on that day. Flying wasn't an option for me then. It must have been a problem for many others too because the conference was not well attended.
At the 2001 conference we reviewed information collected from the industry that said most surveys were still conducted on the telephone and in malls. I recall in 2001 less than 10% of all surveys conducted in North America were via the Internet. Europe was even less.
What a difference six years have made. This year's conference had hundreds more people in attendance and I have never seen so many exhibitors at CASRO in the past. In 2001 Greenfield Online was the only company exhibiting that was talking about using their online survey panel to conduct surveys via the Internet. This year, E-Rewards, Survey Sampling, Research Now, Luth, Greenfield Online, GMI and even a couple of others were there to exhibit their online survey panels.
CASRO now reports that greater than 50% of all surveys are being done on the Web and the conversion to online surveys is happening around the world.
I believe we are entering a new stage of online survey growth. Companies love the speed and access to people at lower costs. Not only are they converting from other methodologies, but they want to do more and more online surveys as a result.
Posted by Dean Wiltse on Wed, May 30, 2007
I recently clicked on this link, "Avoid Pesky Phone Survey People." And this one on Yahoo, "Shut Up Telemarketers for Good."
Did you know that over 50% of customer satisfaction surveys are still being done over the telephone? Isn't that a Double Entendre? Why would a company conduct a "customer satisfaction survey" over the telephone when people hate to get them? Even if a customer is satisfied, the chances are very high that you are going to upset them with a telephone survey. Right?
Millions of people have put their names on the "Do Not Call" list because they don't want to be called. Sure the law allows marketing researchers to continue to bother you because they aren't selling you anything on the call. But come on researchers, wake up. Come out of your academic cave and stop using the telephone for surveys.
Most people don't realize it, but many times the reason researchers go to the phones to bother you is because they are targeting a specific type of individual. And do you know what that means? They are using statistics to find you. In other words, they need to make 100 calls to irritate 99 people to find the right target. Yes, that is the reason. They don't care about the other 99 households that they have invaded, or whether or not you are on the Do Not Call list.
We need to start a movement to put Marketing Research on the Do Not Call list regulations. It should be against the law. Only the lazy and rude researchers continue to use the telephone. Companies and research firms can get people to respond using technology, process and methodology - after all, it is the year 2007.