Free EBook!
We've compiled much of the blog into a free, 73-page ebook, Survey Software Success. The book outlines seven best practices for conducting online surveys.
> Download your free copy
|  |
|
 RSS Feed
Posted by Jeffrey Henning on Tue, Sep 30, 2008
Over and over again I'm hearing researchers complain about questionnaire length. At the AMA MR conference a few weeks ago, at the Gartner CRM conference in early September and at the MRA conference this June, I heard the same complaints:
- "We're making our questionnaires so long that respondents are dropping off half way through the survey."
- "When we have longer questionnaires, respondents become fatigued; they don't pay as close attention to their answers. They rush to the finish; the quality of the results is compromised."
- "When we survey customers, they complain if the questionnaire is too long."
 In fact, in an audience poll at an AMA session, 26% of attendees said researchers need to limit the number of questions they ask, in order to increase online response rates.
So, it's a common refrain, but how do we actually refrain from asking too many questions? The advice tends to be: "Don't ask so many questions!" That's hard advice to follow. Questionnaires, like Congressional bills, have a natural tendency to grow longer. Everybody has questions that they want to include.
So here are some practical tips for shortening questionnaires:
- Answer the Questions Yourself - For customer and prospect surveys, take a hard look at the questionnaire to see which questions can be answered by looking up the data from the appropriate systems. If this is a questionnaire following up to a customer service call, for instance, since your call-center software tracks how long the call was and stores a code indicating the type of inquiry, you don't need to ask the respondent for that information. For that demographic section at the end, your CRM system should have many basic facts about the respondent, such as their age, gender and address. Taking the time to integrate your community feedback system with your other systems can easily cut 10% of the questions you ask, and those are often the most tedious and annoying questions to answer, from a respondent perspective.
- Don't Ask All the Questions - Use skip patterns and randomization to show respondents different subsets of the questionnaire. At the IMC conference, Coast Plaza Hotel staff handed out short paper surveys to luncheon attendees. A few of us discussed the questionnaire, and-to my surprise-while some key questions were the same, the list of attributes to be rated varied from survey to survey. The hotel was able to gather detailed information across many aspects of its services without overwhelming any individual respondent.
- Ask Only the Most Important Questions - A common research tactic is to address a key issue with three or more questions, all with very similar wording. This is important for benchmarks; for instance, the Norwegian Customer Satisfaction Barometer has three questions around corporate image, asking the respondent to rate a corporation's image, asking them to think of how their friends' would rate the corporate image, then asking them to rerate the corporate image when thinking of how well the company does compared to competition. If you're not benchmarking a particular measure, reduce it to one question.
- Don't Ask Questions the Respondent Wouldn't Understand - Sometimes surveys go into excruciating detail about very minor aspects of a product or service. This is understandable, from the perspective of the corporation, which is responsible for every tiny detail. However, many of these distinctions are too subtle for the respondents. The questionnaire might use industry and technical terms that respondents don't know or misunderstand. For cases like this, conduct some qualitative interviews by phone, focus group - or most efficiently - by using your online community; this research can make sure that you are using language the respondent understands and making distinctions that are important to the respondent. Where you are not, get rid of those questions.
- Don't Ask Questions for the Sake of Asking Questions - Look hard at the questionnaire to see if the answer to a question will help you with your current decision making. Sometimes, when writing a questionnaire, you begin to think "It would be nice to know...". How would knowing this particular answer help you? If it wouldn't, then get rid of the question, or-if you really can't stomach that-show it only to a random subset of respondents. For instance, I often seeing pricing questions inserted into surveys that aren't about pricing; these are good candidates to postpone to some later study.
- Survey More Frequently - Finally, if you have many questions on a different topic, then considering running a separate survey on that topic later. The less frequently your organization conducts surveys, the greater the desire to squeeze everything into this survey. By doing surveys more often, you alleviate this. And by using random sampling and enterprise feedback management, you can make sure that you are not oversurveying.
So, how do you make questionnaires shorter? The key secret is to shorten the questionnaire from the respondent's perspective. Hiding questions, showing random subsets of questions and asking questions in a later survey are all ways to shorten this particular survey from an individual's perspective.
Rather than hearing continued complaints about questionnaire length, I hope at next year's conferences that I instead hear people comparing techniques for shortening questionnaires!
Posted by Jeffrey Henning on Mon, Sep 29, 2008
The Christian Science Monitor reported in a blog post, "Obama: Big winner in debate says new poll": "The survey released from USA Today and Gallup show that of the 701 people polled 46 percent believe Barack Obama was the victor [in Friday night's debate] while 34 percent give the edge to John McCain."
In response to this, one commenter, Patrick Ludt, asked, "According to an AOL poll yesterday 52-48 said that McCain won the debate and that was 500,000 thousand people voting how do you explain that?????"
Another commenter, Turk, followed up, with:
701 USA Today/GALLUP................46/34
500,000 MOST LIKELY VOTERS/AOL......52/48
YOU DO THE MATH!
Well, I'll do the math. First, the commenter was misremembering the survey-the AOL debate poll didn't have a half-million responses but had 40,082 responses (at the time of this writing) that went 53% for McCain, 41% for Obama and 6% calling it a draw, markedly different from the Gallup results. Still, I recognize that it is counterintuitive that a study with 40,000 respondents might be less accurate than a study with 700 representatives, but in fact that is precisely the case. Here's why.
To do the math, you have to look at the underlying theory of sampling. The Gallup poll is a scientific, random sample, and can be used to project to the U.S. population. The AOL poll is a self-selected convenience sample, and can't be used to project anything except the composition of AOL site visitors who vote in its polls. A scientific poll needs two main things to be valid: 1) randomness: an equal chance of selecting any member of the population ("probability sampling"), and 2) external selection: respondents are chosen to participate rather than deciding to take the poll themselves. The AOL poll had neither of these things.
 AOL site visitors who vote in AOL polls are not representative of the U.S. population, as illustrated by AOL's recent Presidential poll: 61% voted for McCain for President and 39% for Obama, with 272,939 votes. Just to put that in perspective, if that were the outcome of the election, McCain would outperform his hero, Ronald Reagan, who won 58.8% to 40.6% in 1984. The AOL electoral map has McCain winning 535 votes to 3. No pundit is predicting a record landslide for McCain. So here's a 270,000+ respondent survey that is a poorer predictor of national outcome than the 2,000-respondent surveys independent polling organizations are running.
In case those numbers alone are not compelling, let's try a parable.
A marine biologist receives a grant to estimate the population of fish in a certain area of the sea. He commissions three fishermen to help him. One sails out, drops his net down 200 meters, and pulls up a catch with 90% tuna, 9% swordfish and 1% marine hatchetfish. The second fisherman sails out, drops his net down 1000 meters, and pulls up a catch with 9% tuna, 90% swordfish, and 1% marine hatchetfish. The third fisherman sails out, drops his net down 4000 meters and pulls up a catch with 1% tuna, 9% swordfish, and 90% marine hatchetfish. How many fish of each type live in the sea?
Got me, but I'm a market researcher by background, a fisher of men-and women-in random samples, not a fisher of-fish. All I know is that different types of fish live at different depths of water. The problem with polling is that where you cast your nets determine what kind of results you get back. If you go fishing in red-state waters, you will get red-state voters; if you go fishing in the blue-state ocean, you will get blue-state voters. AOL.com is in red-state waters.
AOL's polls only get AOL visitors. Interrupting people to take a survey, either by telephone or by email, is better than posting a poll on a page and letting people vote. Calling back repeatedly if no one answers the phone and sending scheduled email reminders to take the survey improves the statistical validity by attempting to select the same people more often, to remove any selection bias that might be present if only one contact was made.
Random sampling may not be popular, but it is very accurate. As the old polling joke has it, "If you don't believe in random sampling, next time you go to the doctor for a blood test, have him take it all." You can debate who won on Friday, but you can't debate that scientific polls achieve great accuracy with comparatively few responses.
Posted by Jeffrey Henning on Mon, Sep 29, 2008
 Most of our customers have done a poor job defining member profiles. This is one of the biggest opportunities for improvement that Vovici sees in our clients' online communities, especially as it relates to using communities for feedback.
Community-member profiles should provide all the data needed to achieve three primary purposes: 1) for embedding information to shorten questionnaires, 2) for enabling detailed cross-tabulation and segmentation, and 3) for identifying selected subgroups to invite to surveys and other activities. You should keep each of these in mind when defining what fields you want to store about each community member:
- Embedding Information to Shorten Questionnaires - Don't ask respondents questions that your organization already knows the answer to. Think about how you feel when you have to fill out a contact form at a doctor's office: bored. It's a tedious experience of filling out paperwork. When you write a survey that asks for that type of rote information, you increase the abandonment rate of that survey. Instead, integrate your online community with your CRM system or other sources of such information. Bring over demographic information and data about the member's most recent transaction. For an example of what not to do, see Jim Davies' account of a bad-survey experience with an airline, where this step wasn't followed.
- Cross-Tabulation and Segmentation - Think about how your organization wants to cross-tabulate and analyze survey results and make sure the member profile contains the appropriate demographic and "firmographic" data (information about the organization they work for) to support this analysis. Now take this a step further, as Yahoo! does with their profiles: create fields that are derived from data but that reflect a useful segmentation. Perhaps you want to classify members by the volume of annual business they do with your firm, by how profitable they are or by how likely they are to renew. When you synchronize your community with the appropriate external data source, have your synchronization utility calculate and store the appropriate segmentations for each member.
- Inviting Subsamples - Sometimes you want to invite specific subpopulations of your community to special surveys, focus groups or other community events. Make sure that you are capturing the appropriate information to enable this. These fields should be similar to the fields you need for cross-tabulation and segmentation, but you might want more detail on physical location, past purchases and usage of your organization's services than you defined for your cross-tabs and analysis. This might be an opportunity to incorporate what Bob Page of Yahoo! called "Observed Data", behavioral information such as how often members log-in to the community and how often they participate in forums; these fields can be used when conducting research around improving the community.
The extra effort of integrating these profiles with external data sources is worth it. Surveys are simply more engaging when they are shorter and when the questions are specific and personalized to an individual community member. Properly done, profiles provide members with better survey experiences and provide your organization with a richer source of insights.
Make sure your organization takes advantage of this opportunity to improve your community!
Posted by Jeffrey Henning on Fri, Sep 26, 2008
 Instant coffee is great when you're in a hurry. Of course, it has less caffeine and it tastes more bitter than regular coffee, but it will get you going in the morning.
Likewise, instant feedback is great when you're in a hurry. Of course, it has less accuracy and its findings are more tentative than surveys done over a few days, but it will get you some talking points in a hurry.
Web surveys have long been sold as offering the benefit of "instant feedback", but it's important to remember the caveat. And that caveat is stated fresh for us today in two polls:
- Few in AP poll back Bush market rescue plan. Caveat: "Polls conducted on one night can be less reliable than surveys conducted over several nights because they only include the views of people available that particular evening."
- Americans Favor Congressional Action on Crisis: Four out of 10 say this is biggest financial crisis of their lifetimes. Caveat: "In addition to sampling error, question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of public opinion polls. Polls conducted entirely in one day, such as this one, are subject to additional error or bias not found in polls conducted over several days."
For the purpose of a quick pulse of the American public about their reaction to recent news events, such polls are great. But in the majority of cases, for businesses and non-profits, one night is too short a time to gather feedback. I'll close with my retelling of a story from this year's MRA conference:
Besides the Internet placing price pressure on all forms of survey research, it has also created an expectation that research can and should be conducted rapidly. Robin Pearl of Estée Lauder related a humorous anecdote in her keynote. A salesperson for a panel company excitedly told her how research would be turned around in hours in the middle of the day. To which Robin replied, "Well, I guess I won't be using your company then." The salesperson didn't understand that conducting an entire survey in a few hours would not be representative of the U.S. population, who are not yet chained to their email 24 hours a day but have usage that varies by hour of day and by time zone.
Sometimes good feedback, like good coffee, is worth the wait.
Posted by Jeffrey Henning on Fri, Sep 26, 2008
At the IMC Vancouver conference, Bob Page, Sr. Director, Yahoo! Strategic Data Solutions, presented Insights from Inside Yahoo!'s Data. Given the importance of member profiles in online communities, I was particularly interested in his explanation of Yahoo! profiles.
"It starts with a user profile," said Bob. "A set of features attributed to a ‘user', a user profile contains all features attributed to a given ID." Yahoo! profiles have multiple layers:
- Declared Data - Information provided by the user, such as their gender, geographic location, preferences, social graph and Facebook screen name. I assume the social graph is derived from Yahoo! information, such as their IM buddies in Yahoo! Messenger and their buddies' buddies, as well as from their Yahoo! Mail email address book.
- Observed Data - Behavioral information collected about the user, such as their page views, ad views, ad clicks and searches.
- Insights - Preferences that are implied by their social graph (e.g., preferences of friends) and BT scores.
"In addition to directly targeting customers, we also build look-alike models," said Bob. A target "thumbprint" is prepared for the desired demographic; Yahoo! user IDs are then scored and ranked based on how closely their own user profile's "thumbprint" matches the target. The algorithm takes into account gaps in each user's profile, where data corresponding to the target has not been gathered.
Given that most core Yahoo! services are free, the user profiles also include an estimated "user value". This user value includes the following data, for each user and each Yahoo! web property:
- Finance: Revenues
- Fees from Premium Services
- Non-traditional marketing services
- Traditional marketing services
- Listings
- Finance: Costs
- Cost of Sales
- Op Ex
- Level 1 allocation
- Level 2 allocation
- Data Team: Activities
- Page views
- Ad views
- Clicks
- Feed subscriptions
This user-value calculation (total revenues, total costs, net-value derived) provides a detailed Return On Investment for every Yahoo! user.
Yahoo! user profiles are among the most sophisticated in the industry, and far exceed what is done for most proprietary panels and online communities. Monday, I'll look at lessons from Yahoo! for defining profiles of community members. [Update: Tips for Defining Community-Member Profiles]
Posted by Jeffrey Henning on Thu, Sep 25, 2008
 While my blogger lunch at IMC 2008 was disappointing (only one other blogger at the table), one good thing came out of it: the other blogger told me that John Hossack of VKI Studios, and chair of the conference, was giving a presentation Mapping Visitor Intent Data to Your Clickstream Data.
John discussed how - when it comes to their organizations' web sites - most marketers know "The what people are doing, but not the why they are doing it". VKI has a custom service where they will deploy a popup survey intercept that asks a question:
Respondents' answers to this question are then integrated into the web site analytics of Omniture (today) or Google Analytics (coming soon) or WebTrends (coming later). All reporting is done by a customer's existing analytics package.
The customer can specify the percent of visitors who are intercepted with the question, but John says, "We won't let you intercept more than 90% of visitors, so that the remaining 10% can be used as a control."
Like VKI Studios, Vovici has heard a similar need from our customers, and from attendees of the IMC show and the MRA show in New York City this summer. We've packaged this into a product called WhyClicked, for which we've prepared a Flash intro:
If you don't like Flash, here are our WhyClicked screen shots.
We're glad to see VKI Studios evangelizing this concept. Behavioral analysis from web analytics is great, but integrating it with attitudinal analysis helps complete the picture of your site visitors.
Posted by Jeffrey Henning on Tue, Sep 23, 2008
This afternoon at ESOMAR, we announced a new partnership with TNS:
 TNS will adopt the Vovici EFM Platform and Community Builder Module as its standard platform for building custom online panels and communities for its customers.
Over the past 5 years, the TNS Custom Panel team has built qualitative, quantitative and mixed-use custom (proprietary) panels for consumer and B2B audiences, capitalizing on TNS's 60+ years of experience in building, managing and maintaining panels globally. Custom panels are not new. What is new is the emergence of Web 2.0 and, in particular, the familiarity and wide-spread use of social networking platforms.
"The end-to-end solution we are offering with this partnership is unique and really powerful," said Leslie Warshaw, Vice President of Proprietary Panels at TNS. "TNS research and panel expertise together with the Vovici technology platform delivers an industry leading solution enabling us to build the most flexible and comprehensive custom panels and communities for our clients."
I'm very excited by this announcement, as Taylor Nelson Sofres was our first customer back in 1993: only then they were known as Taylor Nelson, and we were known as Perseus Development Corp. If I had the good sense to frame the first dollar the company made, it would have been a British pound note from them. Back then, we were providing Taylor Nelson interviewers computer-assisted personal interviewing software, which was the modern tech of the day. Now, we're providing our feedback-community software. This is a great opportunity for us to build a new partnership, given the many years of experience TNS brings in custom panel management.
Posted by Jeffrey Henning on Tue, Sep 23, 2008
 Yesterday I described the "Twists and Turns on 21st Century Internet Express" presentation (The Survey Taker is King) at the AMA 2008 Marketing Research Conference. Many of the problems that face traditional market-research surveys go away when a corporation is able to survey its customers, employees or resellers, rather than a third-party population. Enterprise-feedback management systems and feedback communities alleviate many of these concerns.
The audience was polled for their concerns with the question, "What should be done to increase online survey cooperation rates?"
- Make surveys more engaging (50%) - Actually, the most engaging surveys are those for relationships that the respondent has the greatest involvement in. The challenge for MR is getting people they have no relationship with to take a survey. Employee surveys typically have the highest participation rates, because employees invest a lot in their work; customer surveys have solid response rates, with greater response rates the more money is spent; surveys of prospects or general consumers have the lowest response rates. When you're surveying your community, you start with engaged survey takers; your job is to respect that engagement and not dilute it through bad surveys.
- Limit the number of questions/time to complete (26%) - Traditional market research surveys tend to be long: researchers know nothing about the respondent and need to ask them detailed demographic and firmographic questions; researchers will often ask a question multiple ways to develop a more reliable index that can be used for tracking; researchers will have many grid questions, gathering different aspects of attitude for a common list of topics. Many of these reasons for length go away for the community researcher: demographic and other profile information can be piped into the survey, rather than prompted for; individual key questions are asked rather than index-question groups; finally, researchers can survey often, rather than getting every answer from this single survey.
- Better target surveys to individuals (15%) - Using the profile data in a community or enterprise feedback management system, community researchers can invite the right respondents, where market researchers need to invite everyone, then ask questions to screen out the wrong respondents.
- Increase the use of incentives (7%) - Incentives are less necessary in a feedback community, where respondents are more motivated by their engagement than by extrinsic rewards.
The reverse of the question was also asked to session attendees: "What factors are contributing to lower online co-operation?"
 After "Length of time required to complete surveys" at 51%, "Frequency of invitations issued to survey participants" (30%) and "Lack of online panel development and active management" (11%) and were the two most popular choices. EFM systems are all about touch management, making sure community members and panelists are not oversurveyed. Internet panels, used for generic market research, have forsaken the traditional limits on survey participation, and are further hobbled by the need to invite people to take a screener survey before qualifying for the full questionnaire.
Clearly, for researchers who need to survey their organization's key constituencies rather than the market at large, enterprise feedback management systems and feedback communities eliminate many of the pressing challenges faced by market researchers today.
Traditional researchers haven't needed to be responsible to the respondent, who was a replaceable commodity; for community researchers, respondents are valued customers, employees and partners, and therefore should be treated like royalty.
Posted by Jeffrey Henning on Tue, Sep 23, 2008
 As a survey wonk, I love the Presidential election cycle. Every four years a tremendous amount of attention is paid to polling and survey research, as candidates and the media try to understand the horse race for President. I learn a ton about the latest techniques and challenges, as research techniques are applied to a unique problem. For the rest of this week, I am going to write a post a day about polling trials and tribulations and some ramifications for traditional research.
Inspiration for this series is last Wednesday's keynote to the AMA 2008 Marketing Research Conference, What Marketing Researchers Can Learn from Polling Conducted for Candidates and the Media. Humphrey Taylor, chairman for the Harris Poll presented.
Humphrey's nine key points:
- Being a good researcher is necessary but not sufficient.
- Understanding motivation is very difficult - but vital.
- Forecasting requires both good research and good judgment.
- Balance theory and empiricism.
- Balance trade-offs between quality, speed and cost.
- Understand what qualitative research does well and badly.
- Run scared.
- Continuous quality improvement: the need to change and improve methods and application.
- Research is no substitute for creative thinking.
Political polls are asking people about something that they might do, but haven't done yet: vote. In this way, the lessons of political polls apply to surveys about intent: intent to purchase, price sensitivity (intent to buy at different price points), willingness to recommend, etc. Humphrey's next points apply equally well to consumers asked to articulate their next planned purchase as it does to citizens asked to predict their vote:
- People do not understand themselves.
- They rationalize their differences.
- They deceive themselves.
- They lie.
A common staple of survey research is the list of attributes that you rank products on. Yet buyer motivation, like voter motivation, is very complicated (Humphrey lists ten items that affect voter motivation, from party identification to personal media consumption).
I don't think I've ever conducted a study where respondents didn't end up dramatically overstating purchase intent. People are cooperative with the interviewer (whether in person or online) and overestimate their likelihood to purchase. Similarly, when polls go awry in predictions, it is often for overestimating the voter participation rate of key demographics.
In fact, that's just one of six factors that can cause polls to make the wrong prediction. Here are Humphrey's points for polls and my translation of those to survey research in general:
- Differential Turnout - Just like voter blocks sometimes don't go to the polls as predicted, sometimes customer segments don't make it to market in the proportions you predicted (maybe your channels don't align with the targeted segments).
- Differential Response Rates - Sometimes your respondents do not reflect your target population in the right proportions. Make sure to factor that into your analysis.
- Bad Methodology - Common sampling techniques are probability samples, non-probability but weighted samples, and convenience samples. Convenience samples (posting a poll or a link to a survey on your home page) should never be extrapolated to represent your target audience; whether or not you should do that with non-probability weighted samples is a matter of argument (the Harris Poll is based on a non-probability sample weighted for key demographics).
- Sampling Error - Sometimes the prediction isn't wrong, but was inside the range of sampling error. The more important the decisions that are going to made based on the results of your survey, the more important it is to have a larger survey sample, thereby reducing the range of sampling error.
- Measurement Error - "Sampling error" is just the most well known error, receiving the most attention because it is the most easily estimated. Measurement error is just one type of non-sampling error: Different scales can produce different results, poorly worded questions can affect measurement, and so on. The more important the survey, the more attention needs to be paid to every detail.
- Late Swing - Sometimes market conditions change. If a major event has occurred since your survey (for instance, a competitor has dramatically lowered prices or released a heralded new product), you may need to update your research before drawing any conclusions from it.
For a take on this topic from the other side, see What Political Polls Can Learn From Market Research Surveys, an MRA presentation given in June by Kathleen A. Frankovic, director of surveys for CBS News. AMA's Marketing News Blog has a post on Humphrey's presentation, as well.
 Oh, and don't be too hard on that famous Dewey Defeats Truman headline that I used to illustrate this article. For two reasons. First, when elections are "too close to call", they are, well, too close to call. Switch a few thousand votes in Ohio, Illinois and California (differences all too small to see with polls) and Dewey would, in fact, have won. Second, the reporter for that story was one Arthur Sears Henning. As the Chicago Tribune put it, "Maloney banked on the track record of Arthur Sears Henning, the paper's longtime Washington correspondent. Henning said Dewey. Henning was rarely wrong." (According to a poll of my relatives, he's no relation, but I could be wrong on that.)
Posted by Jeffrey Henning on Mon, Sep 22, 2008
Would you devote your home page to soliciting new ideas from your customers?
Oracle would. Oracle is. The new Oracle community, built with Oracle Mix, is immediately accessible from the Oracle home page. You can even submit an idea without logging in. Of course, if you log in, you will be able to vote on ideas and participate in discussions about those ideas.
The new community site will only be featured like this on the home page for a week, but it's a great way to make a splash.
(Jim Storer posts good ideas for improving member on-boarding for the new community.)
Reminder: if you are using Oracle CRM On Demand, our upcoming connector will easily synchronize your CRM data with your Vovici EFM system.
Posted by Jeffrey Henning on Mon, Sep 22, 2008
One of the panels at the American Marketing Association's 2008 Market Research conference in Boston was Twists and Turns on 21st Century Internet Express. Panelists included David Santee, director of market research at H&R Block; David Shanker, COO of OTX Research; and Michael McCrary, a vice president at Greenfield Online, sitting in for Keith Price.
David Santee said:
Our theme is "The survey taker is king!" - on the client side, it's not taking too much of a leap to say that we don't think about this guy too much. We have to balance 15 projects, work with providers, try to be a consultant to senior management; that's a lot of stuff, so the respondent gets the shaft in terms of our mind share.
There are many initiatives about data quality, but we aren't addressing those, because we want to focus on the survey taker. It is our collective agreement that the more we can increase the quality of the survey experience and make the survey fun and compelling the more we can help with data quality.
The panelists conducted an instant poll: "What percent of your research is completed online?" Surprisingly, 32% of audience members said 80-100% and 25% said 60-79%. Only 17% said 0-19% of their research was conducted online.
Michael said:
To do a lot of online surveys, we need a lot of online survey takers. It is all of our responsibilities to make sure that we engage these respondents, so that they are engaged and cooperative and give us better responses. It is our job to engage as many people in the survey research as possible.
We do enough satisfaction surveys after surveys to know what works and doesn't work. As service providers, no matter how many times we talk about guidelines, those are not always followed. Again, doing a survey for more than 20-25 minutes is a bad idea. That 45-minute survey - we know there are times when we need to do that - will get a lot of drop-offs and will take more time to fill.
We need to come to a consensus. If we don't engage survey takers, we are going to lose that growth curve.
We [at Greenfield Online] are not competing with other sample providers but with people's other uses of the Internet. When we started, email was unique and new, and our surveys were cool and hip, as was anything on the Internet. How times have changed! Now it's "text me" or "FaceBook me, don't email me".
I try to draw correlations to taking an exam, like taking a web survey. It gets hard to focus after 40-45 minutes, asking them to do things for a long time. The Internet is an evolving communications platform For the most part, we've just taken forms and put them online; we are not using the innovative creative techniques out there. Respondents are frustrated with answering mind-numbing surveys, with answering grid questions.
The audience was asked to respond to two polling questions:
 
[Audience members could make only one choice for each of these selections. Approximate sample size was 200. Results are only representative of AMA MR conference attendees.]
David Shanker said:
Surveys should be shorter; no amount of making it more fun is going to make hour-long surveys enjoyable.
Lots of straightlining or inconsistent answers shows where frustration sets in. Some questions are confusing. This ups the importance of writing a questionnaire that is friendly.
David concluded by asking, "Do you take your own surveys?" To the poll, 77% (134) of responding audience members said that they did take their own surveys, 23% (40) said that they didn't. The panelists were surprised at how many people were taking their own surveys.
Panelists recommended the following best practices:
- Respect the survey taker
- Take your own surveys
- Clearly define the value proposition to the survey taker: "You have to compare surveys to other experiences they have online; for instance, young men play games."
- Do not ask the same question over and over: "Minimize the repetitiveness. I know, the first time is to establish interest, the second time for pricing, the third time for purchase likelihood. They get to that third battery, and they do not want to do it anymore."
Michael again:
The demand [for online panel] has increased dramatically, but the supply has not increased; the price has gone down, but the demands for quality have gone up; all those things have happened at once, which is illogical.
What is the value proposition for the survey taker? As you build communities, as you build panels, it is not a financial windfall to join these, so it has to be about the experience. Define the fact that when they take the surveys it is for a good reason: it affects the products they buy, the commercials that they see, those things resonate with people in ways the financial incentive doesn't.
For another attendee's take on the presentation, check out Josh Mendelsohn's recent post about it.
Update: For my own reaction to this presentation, see Treating Your Survey Takers like Royalty.
Posted by Jeffrey Henning on Fri, Sep 19, 2008
 On Monday, Monica Wood, VP of Global Marketing Services for Novartis Consumer Health, gave a great presentation at the American Marketing Association's 2008 Marketing Research convention in Boston, entitled Delivering Continuous Insights: Involving Consumers in Every Decision. Monica began by saying, "New technologies in market research bring the consumer and key influencer into the meeting room at every business decision point. Online communities allow market researchers and internal teams to deliver continuous insights through near ‘real-time' interaction with target consumers and professionals."
Novartis adopted online communities in the interest of saving research time and research dollars, while increasing the frequency of research. Novartis typically uses its communities for qualitative research, "deliver[ing] early insights in a timeframe allowing concept and program refinement prior to quantification".
Online communities for Novartis work best when private and secure, with participation by invitation only, after rigorous screening of potential invitees. Novartis targets about 300 members per community, with some communities representing a narrow demographic and some communities designed to represent the broad target population. In the third quarter of 2006, Novartis launched a consumer community, a professional influencer community and a pharmacist community; these were so successful that Novartis launched two additional communities in the last quarter of 2007. Novartis uses the communities for understanding brand insights, developing packaging, estimating the strength of clinical messaging and gauging the impact of promotions on purchases.
For recruiting, Novartis used standard recruiting methods that are used for building panels, involving online lists, with telephone screening interviews to make sure the people were who they said they were. Members are typically recruited for six-month periods, but some are kept for a year.
Similar to enterprise feedback management initiatives, online communities need "rigorous partnering with internal teams (i.e., privacy office, IT, legal team)" to ensure compliance with privacy requirements, information security practices and incentive administration.
Surveys are an important part of online communities, needed to obtain qualitative feedback from the entire community and providing opportunities for "structured routing for questions" and for demographic cross-tabulation. Member demographics and practice firmographics (size of the doctor's practice, age of the practice) are used in breaking out the survey results.
Besides surveys, the online community provides the following tools: discussion threads, online chats (analogous to focus groups), gallery of member-posted images, and more. For a specific need to develop materials for doctors, the team used most of these tools to develop messaging collateral prior to quantitative testing:
- Discussion forums to understand the information used to select a treatment for the condition
- Gallery to share images of the types of materials that doctors found informative
- Online chats both to obtain initial reactions to early drafts and to obtain final reactions prior to formal quantitative testing
- Surveys to gather feedback from all the doctors
The members of the communities become very engaged partners. Monica said, "We ended one community but people actually went crazy. They were ballistic, saying, ‘we love this, don't close it!' The doctors didn't want us to stop. The consumers didn't want us to stop. Once you start a community, be willing to invest and commit to it, as you may get a backlash if you try to close it down. You have to close it down gently and amicably."
In the past year, Novartis has conducted 130 community surveys, 330 discussion threads, 160 brainstorming activities, 14 online chats and several diary assignments. Novartis considers online communities to be a "Best In Class" research approach at "half the cost of traditional methods" with "months time savings".
Posted by Jeffrey Henning on Fri, Sep 19, 2008
 So when I was looking over presentations to attend at the Gartner CRM show, one of the presentations I chose was Social CRM Meets the Enterprise. I assumed it was a Gartner presentation, and once there was pleasantly surprised to find out it was an Oracle presentation, as we were in the middle of extending our partnership with Oracle.
Yesterday we announced that Oracle has chosen us as a CRM On Demand Inner Circle Partner. Here's Julie Adams, Oracle Vice President:
Our product management team has thoroughly reviewed the partner ecosystem for the most synergistic solutions from our partners. We believe Vovici, as part of the Oracle CRM On Demand Inner Circle Partner Initiative, will provide our customers and prospects complementary value-added functionality in the enterprise feedback management arena.
We've been integrating our EFM systems with CRM systems since 2004, back when most of our clients were running proprietary CRM applications. In 2004, we had a small suite of perhaps a dozen SOAP web service calls that could be used for integration. By 2006, that had grown to an extensive layer with over 100 web-service calls. Now, with the announcement of the Vovici CRM 2.0 Connector for Oracle CRM On Demand, we have taken a few weeks of custom development effort and reduced it to a few days of mainly data configuration.
Why integrate your enterprise feedback management system? To treat your clients as if you actually know who they are when you survey them, to minimize the amount of information they must provide, and to avoid bad survey situations like the one Jim Davies of Gartner related.
Why integrate your EFM system with Oracle CRM On Demand? Because now, not only is it the right thing to do, it's the easy thing to do.
Posted by Dean Wiltse on Thu, Sep 18, 2008
 Two weeks ago, Jennifer Jones, of the MarketingVoices podcast, conducted a short interview with me about the rising importance of communities. You can listen to the podcast from your browser here:
Community Spirit - Building Sites in the World of the Corporate Enterprise
It's an exciting time in the industry for communities. Every marketing person I talk to wants to do one, has just started a community, or is looking to grow their existing communities. But all of this reminds me of the early days of the Web, when every company wanted a web site, but wasn't sure how they would use it.
Not sure how you can use a community? Download the podcast or listen through your web browser, then give us a call at +1 703 481 9326, or drop us a comment here.
Posted by Jeffrey Henning on Wed, Sep 17, 2008
 Some of my impressions of the 2008 Internet Marketing Conference, as seen through my community-colored glasses.
Patrick Schwerdtfeger of Tactical Execution defined "social media" as "platforms that facilitate communication between users". He discussed five types of platforms:
- Social networks - MySpace, Facebook, LinkedIn
- Social bookmarking - Digg
- Microblogging - Twitter, Utterz
- Sharing communities - Flickr, YouTube
- Group platforms - Yahoo! Groups
I would imagine that Vovici Community Builder and its like would be included under the "group platforms" segment.
One presenter said "Social media-It's not about the media". Patrick doesn't like either word of the label: he said that calling it "Web 2.0" is passé, and calling it "Social Media" as we now do is also passé, because it really is "Community Engagement".
Beyond Site Analytics
I heard a few times that when it comes to understanding your web visitors, site analytics are not enough, behavioral data is not enough: you must include attitudinal data - survey data - as well. This was mentioned by Jason Burby of ZAAZ in the Web Monetization presentation and by John Hossack of VKI in his presentation on site-visit intent (which I hope to blog about soon; update: see Mapping Visitor Intent Data to Your Clickstream Data).
Tips for Writing Survey Invitations
Monique has some good tactical advice for writing email newsletters, advice which I thought was also useful to keep in mind when writing survey invitations. Here they are, as blogged by Miss 604:
- Avoid computer-generated/anonymous return email addresses.
- Have a catchy subject line, keep it short, less than 50 characters (ideally 25). Be descriptive and never leave it blank - that just makes it spam filter bait.
- Have flow between your subject line and your headings within the email.
- Scanability - people don't have a lot of time to read a long email so make sure they can "scan" the newsletter and pick out the content. Make it look appealing, have headers, catchy lines etc.
Blogs
Blogs weren't as popular as I expected. One panelist on the social media panel even admitted to not yet having a blog. And, at the Thursday lunch, a "Birds of a Feather" event where tables were organized by interest, I was shocked to be one of only two people to sit at the "Bloggers" table.
That made for a short conversation, and I decided to spend the rest of the lunch period walking along the sea wall at Stanley Park, which was absolutely beautiful. It was just minutes from the hotel and offered scenic views of islands, forests, boulders, cityscapes and monuments. I couldn't help but wonder if the totem poles at Brockton Point weren't the social media platforms of their day...
Posted by Jeffrey Henning on Tue, Sep 16, 2008
 At the AMA 2008 Market Research Conference in Boston yesterday, two of the presentations I attended discussed generational differences. J. Walker Smith, president of Yankelovich, focused on Boomers with From One Generation to Another: A Yankelovich MONITOR Perspective on the Resurging Marketplace Importance of Cohorts, and Brad Karsh of JB Training Solutions focused on Millenials with Dude, What's My Job? Managing Millennials in Today's Workforce.
Here's Smith's segmentation of generations:
|
Matures |
Boomers |
X-ers |
Echoes |
|
1909-1945 |
1946-1964 |
1965-1978 |
1979-1990 |
|
52 million |
78 million |
57 million |
51 million |
Both speakers did a good job at presenting the differences between generations in an entertaining way, from the serious to the silly. Smith again:
|
|
Matures |
Boomers |
X-ers |
Echoes |
|
Their War |
World War 2 |
Vietnam |
Gulf War |
War on terrorism |
|
Their Doctor |
Dr. Spock |
Dr. Strangelove |
Dr. Kevorkian |
Dr. Phil |
(As an older X-er, I would have said my doctor was Dr. McCoy.)
Here are some stats that contrast Boomers and Generation X-ers, showing X-ers' greater focus on the financial:
- In 1974, 51% of Boomers were interested in saving for retirement; at a comparable age, in 1998, 73% of X-ers were interested in saving for retirement.
- Where 63% of Boomers were concerned "about understanding my own reasons for doing things", only 27% of X-ers were in 2002, at a similar age.
- In 1967, as incoming college freshman, 82% of Boomers were going to college to "develop a meaningful philosophy of life" (the number one reason); in 2003, that was the last reason for X-ers, 74% of whom said "being very well off financially" was why they were attending college.
So, a compelling case can be made for looking at survey data by generations. I hope some attendees went back to the office and wanted to analyze some of their surveys to tease out these generational differences in their own industries. Unfortunately, too many times, that can't be done, because the age question used won't support it.
The following is a popular type of demographic question for age, this one taken from the profile questionnaire for the American Marketing Association itself:
Age Range
[ ] 21 and Under
[ ] 22 to 34
[ ] 35 to 44
[ ] 45 to 54
[ ] 55 to 64
[ ] 65 and Over
[ ] Decline
Moral of the story: If you want to be able to analyze your survey data by generational cohorts, make sure to ask the respondent for their birth year rather than their age range.
Posted by Jeffrey Henning on Tue, Sep 16, 2008
 At the Internet Marketing Conference 2008, William Azaroff of Vancity (Canada's largest credit union) gave a great presentation entitled Engage Community With Your Brand. His three main points:
- All social media is inherently authentic
- The first 500 community members set the tone
- Think of community engagement - not social media.
TechVibes.com has a good summary of his presentation. A key point, for those of us building feedback communities:
In general, the first 500 members of your online community will set the tone and mood. If the first 500 registered users are yelling, being rude and creating havoc, then the people you really want to participate aren't going to feel welcome. Fortunately the opposite is also true. If you personally choose your pioneer users and have them pass along the URL to like minded peers, you will being to attract more people who get what you're trying to achieve. Once you hit this milestone and the masses get hold of it, the foundation will be set and it'll be far easier to monitor and moderate. In fact, you may find that you don't have to moderate comments much at all, which is what the case has been with ChangeEverything.ca.
Posted by Jeffrey Henning on Mon, Sep 15, 2008
 I got into Vancouver late Wednesday night for the Internet Marketing Conference. Being driven from the airport by the cab, I looked out at the city, and it seemed like any other big city at midnight. Thursday morning, I opened my curtains and literally gasped in delight. I was on the 34th floor of the Coast Plaza Hotel, and I had a breathtaking view of ocean, islands, mountains, bridges and cityscape. Below me was a block of tall apartment buildings, followed by forest, beaches, harbor, lagoon: Stanley Park. Top it off and it was unseasonably sunny and warm. Vancouver demos well.
I'm already off to the next conference (AMA MR, in Boston), so for now I will point you to other people's summaries of the conference:
Tomorrow I will highlight some excerpts of the conference that I found particularly relevant to feedback communities.
Posted by Jeffrey Henning on Mon, Sep 15, 2008
 Summary of my coverage of the 2008 North American convention:
Next up, some excerpts from the Internet Marketing Conference.
Update (10/09/08): Feedback survey on the conference champions "green" web surveys.
Update (10/19/09): Gartner CRM Conference 2009 Recap.
Posted by Jeffrey Henning on Sun, Sep 14, 2008
 At the Gartner CRM Summit 2008, Anthony Lye of Oracle presented Social CRM Meets the Enterprise. Some quick points on his presentation follow.
Anthony began by differentiating CRM 1.0 and 2.0 by focus and strength:
- CRM 1.0 - Transactions - understanding hierarchy
- CRM 2.0 - Conversations - understanding social networks
He characterized the rise of social media as the biggest disruption that has occurred to CRM. (Gartner itself avoids the term CRM 2.0, which means different things to different people; see CRM 2.0 = CRM + EFM.)
Traditional CRM doesn't understand conversations, but social networks don't understand business. More than 40% of business incomes happen outside of any one enterprise.
How important are conversations? If you perform Google searches on the top 20 global brands, 30% of the links returned on the first page for each brand are links to social-media sites. This was probably the most important statistic I learned at the Gartner summit, as I've already found myself repeating this stat a few times this week. Clearly, the crowd is shouting down the voice of the brand.
Anthony also cited Edelman's Customer Index Report 2008: 76% of respondents would rate a person like themselves as the most credible spokesperson for a brand, significantly greater than the amount that would choose the CEO. While the Internet may account for only 10% of total U.S. sales, online social networks are estimated to be impacting 40% of U.S. sales.
The terms "online community" and "social network" are often used interchangeably, but Oracle recognizes three types of social network:
- Consumer Social Networks
- Productivity Social Networks
- Customer Social Networks
Customer Social Networks are vendor-centric, enabling members to discuss business processes with fellow customers.
History, according to Oracle, can be divided into three epochs:
- 1980s - On Premise solutions - IT innovates
- 1990s - On Demand solutions - the Line of Business innovates
- 2000s - Social solutions - the "edge" innovates
I'd quibble with the timing (on-demand solutions have really come into their own only in the last five years) but I agree with the broad strokes.
Oracle argues that selling is fundamentally social, but that SFA is about reporting more, and selling less, as a result of an operational effort of pleasing management rather than customers. The Alexander Group surveyed 1,000 North American businesses and found that only 22% of an account executive's time is spent selling; most is spent in administration and reporting; only 3% of time is devoted to selling new products. Truly social CRM allows account executives to report less, and sell more, becoming a sales productivity application.
Oracle is applying social media to a host of Oracle Social CRM applications, including Oracle Sales Prospector, Sales Campaigns, Sales Library, Spaces and Sales MIX.
Posted by Jeffrey Henning on Sat, Sep 13, 2008
 At the Gartner CRM Summit 2008, Gartner analyst John Radcliff presented The Revolution in CRM Architectures and Technology. His "Aha!" slide indicated the following top-ten priorities for organizations to consider this year (based on a survey of Gartner analysts):
- Web 2.0
- Cost/Growth/Innovation
- SaaS
- Modernization of applications
- SOA/SOBA/SODA/EDA/BPP
- CE/CRM
- Application governance/PPM
- Business process discipline
- Application architectures
- MDM/EIM
For Web 2.0, John discussed communities and social networks, saying "the digital native has arrived". Web 2.0 will have an increasing impact on CRM, as digital natives ("Generation V", those who have grown up with the Web) become customers and employees. Online communities now provide consumers with a way to make their voice heard by leading brands.
For the move to SaaS (Software-as-a-Service), John discussed the significant growth in on-demand CRM suites. (For us at Vovici, our software sales have declined even faster than we budgeted for, due to the significant shift and increase in our SaaS sales.) While there are contrarians out there, SaaS is winning over more and more organizations.
Enterprise feedback management, while not explicitly included in the Top 10 (next year!), was included, and mentioned, within CE (Customer Experience) technologies.
For mobile devices, John pointed out that handsets are driving the Mobile Web: there are 200 million notebook computers with excellent browsing experiences, but 1.5 billion handsets, with 35% providing good web browsing experiences and 50% providing reasonable browsing experiences.
John discussed the evolution from UIs to UXs, from user interfaces to user experiences, granting users ever greater control of their applications, with wider accessibility to the applications from multiple platforms.
He concluded with evaluations of each of the major CRM vendors, and how well they achieved SOA, either as a wrapper around existing applications or as a fundamental redesign of an SOA platform.
John had a lot of good content, but if you were playing "buzzword bingo", you would have your had best chance of winning during his presentation!
Posted by Jeffrey Henning on Fri, Sep 12, 2008
 At the Gartner CRM Summit 2008, Gartner analyst Gareth Herschel discussed Customer Service Analytics: Adding Intelligence to the Service Experience. He outlined four best practices:
- Define scope of analysis: Is your organization analyzing the service process in its entirety or the IVR/call-routing system or the call center staff?
- "Good, inexpensive call" may be an oxymoron: A call that is too long costs more and annoys the customer; a call that is too short may be missing an opportunity for engagement with the customer and may not resolve the issue at hand.
- One size doesn't fit all: Don't judge all calls by the same criteria. Categorize calls by the purpose of the call and the value of the customer, and then develop metrics for this segmentation.
- Tailor results to different views: Too often analytical reports don't really meet the needs of anyone in the organization. Some reports need to focus on "exceptions not averages", and should include alerts and triggers so that the company can take action to intervene on behalf of dissatisfied customers; some reports need to focus on the big picture, providing "context not summaries" and should try for advanced visualization. Too often dashboards have become just a collection of metrics; the best practice for dashboards is to compose them of "related metrics that paint a whole picture when presented adjacent to one another."
Gareth made the point that companies should pay special attention to "bad news". He quoted Despair.com's service department: "We're not satisfied until you're not satisfied." He said, while it was tongue-in-cheek for Despair.com (a publisher of "demotivational" posters), it was painfully true of many firms, who - too often - make providing negative feedback an arduous process, "putting the complaining customer into a penalty box". Negative feedback can often lead to the best insights.
For capturing market insights from service interactions, Gareth displayed the following quadrant analysis:
|
Mature Analysis Strategies & Techniques |
Immature Analytical Expectations & Best Practices |
| Data We Need to Collect |
EFM |
Blogs |
| Data We Already Collect |
Enterprise Data Warehouses |
Customer Call Recordings |
Blog and forum feedback can be "unreliable, prone to manipulating, and unquantified". Gareth suggested using it to identify potential service issues that were then quantified using enterprise feedback management.
Analysis of customer-service issues can be applied in three different ways:
| Concept |
Tactic |
Role of Analysis |
Analytical Techniques |
| Pre-emptive: Customer Issue Avoided |
Identify & Resolve Causes of Problems |
Identification of High Cost Issues |
Cost Allocation
Call Categorization
Root-Cause Analysis |
| Pro-active: Customer Notified Issue is Being Addressed |
Resolve Issue on Corporate Schedule |
Identify Treatment Strategy |
Issue Detection |
| Re-active: Customer Calls to Resolve Issue |
Divert Issue to Self-Service Channels |
Divert Issue to Self-Service Channels |
Intelligent Call Routing |
Gareth concluded with a complex slide showing categories of service analytics vendors. He pointed out that no suite solutions exist yet. Over time, different technologies will become bundled together and consolidation will happen. He said that enterprise feedback management and text analysis were a natural fit, and he expects more bundling of such solutions in the future (Clarabridge and Vovici have been doing just that since July). He doubted that 10 years from now the dozens of technologies he showed would be bundled into a standard customer-service analytics suite. He said, if it did happen, there would be another 20 new categories of analysis that would have emerged and would not be bundled in the suite!
Posted by Jeffrey Henning on Thu, Sep 11, 2008
 On Monday, at the Gartner CRM Summit, Jim Davies gave the presentation The Role of Technology in Improving the Customer Experience. He began with a compelling point: billions have been invested in CRM systems, yet customer satisfaction has not improved. How to explain this discrepancy? Most CRM systems have been implemented from an operational point of view, not from the view of customer experience. In fact, many technologies have made the customer experience worse! Annoying IVR (Interactive Voice Response) systems, hard-to-use web sites, poor call routing, call centers with powerless customer service agents all save their companies money, but at a cost to the customer.
Clearly, technology needs to be used to improve the customer experience, not just to achieve operational efficiency. How to reconcile these two competing goals? Jim proposed a compelling analytical framework for prioritizing which technologies to adopt. He presented a series of charts graphing investments on the dimensions of Productivity (how much does this technology improve customer service productivity?) and Impact (how big of an impact does this have on a customer's experience?).
Each technology is then plotted with a colored circle indicating ease of achieving ROI: green for easier, yellow for moderate, red for challenging.
The composition of the actual charts will vary for your industry and organization.
Key to improving the customer experience is the need to better collect feedback and better analyze opinions. Jim outlined the following technologies to help:
Collect Feedback
EFM (Enterprise Feedback Management)
Survey tools
Event analytics
BPM (Business Process Management)
QA monitoring
Analyze Opinions
Customer value analytics
Data mining
Segmentations
Sales/CSS/Marketing
Jim made another key point: balance customer surveys with detailed analysis to determine what customers really want, in order to invest in the appropriate improvements. Jim gave the example of BT Retail, which was trying to determine the lack of correlation between satisfaction scores and intermittent failure to meet SLA (Service Level Agreement) response times. For certain periods, BT field-service representatives were not showing up at customer locations within the committed time window. Yet for those transactions, satisfaction was no lower than for transactions that were within the time window. Meanwhile, BT was expending significant resources to try to improve the responsiveness. From further research, BT determined that the SLA lapses were occurring during periods of difficult local weather; customers were happy that a BT representative showed up at all, given the conditions, and therefore did not penalize the company for the delay. As a result, BT shifted its efforts to better communications and settings of expectations, by calling affected customers in advance to let them know that the weather would result in a delay.
Jim's final recommendations for improving customer experience:
- Create technology matrices to prioritize which technologies your organization should invest in to improve the customer experience
- Look for appropriate opportunities to embrace existing technologies
- Focus on improving important attributes
- And finally: "Build an EFM system and explore more innovative feedback mechanisms."
Posted by Jeffrey Henning on Wed, Sep 10, 2008
Yesterday I covered one slide of Jim Davies' presentation, EFM: The Who, When, Why, Where & What of Surveying. Today I wanted to look at some of the highlights of his presentation.
The What? Jim described EFM systems as the feedback "hubs" of their organizations, the central repository for survey data.
Why? Without EFM systems, organizations trip all over themselves, oversurveying customers and missurveying them, because of a lack of attention on process and results. Check out yesterday's post, "When Surveys Go Bad", for four negative case studies Jim presented of what happens when organization don't adopt EFM systems.
Who? Gartner mentioned four firms as leaders in enterprise feedback management: ConfirmIt, Vovici, SPSS and Ransys. He also mentioned other companies, categorizing them as low-end tool vendors or as process vendors, focusing on one narrow part of feedback. Two "companies to watch" for Gartner are Cvent and Fizzback, with its novel SMS solution.
When? Begin planning for EFM implementations now! Gartner estimates that EFM implementations will grow 15-20% in 2009 over 2008. When planning to implement an EFM system, Jim noted that "EFM functionality is more diverse and complex than many organizations perceive." Regarding on premise vs. on demand, Jim said, "Don't be afraid of SaaS, don't avoid it out of fear, but do look at the complexity of integration."
Where? By department and across the organization. In online communities. Jim pointed out that EFM can be used for "extracting insight from the chaos known as social computing". To accomplish this, set up an online community with member profiles containing information such as the member's product portfolio, demographics and geographic location. Use forums for ad-hoc feedback, with text mining supporting the qualitative analysis. For the quantitative side, field surveys to not only community members but to customers who aren't community members as well; let customers select their preferred modality. And my favorite quote from the conference: Jim Davies said, "Only one vendor does all this - Vovici." Otherwise, he pointed out, you should expect to find an EFM application and integrate it with a separate community platform. Of feedback management systems and social community platforms, Jim expects that "these realms will merge."
You can purchase a recording of EFM: The Who, When, Why, Where & What of Surveying, and all the other Gartner presentations from the Gartner Customer Relationship Management Summit 2008, at the Gartner event site.
Posted by Jeffrey Henning on Tue, Sep 09, 2008
Jim Davies, an analyst with Gartner, gave a great presentation today in National Harbor, MD, at the Gartner CRM conference, entitled EFM: The Who, When, Why, Where & What of Surveying. He started with four great anecdotes about surveys gone awry.
Lengthy Surveys - One day he had car trouble, and his auto club towed him home. The firm made the best of a bad situation for him. He was very pleased with the service, until the driver handed him a PDA and asked him to fill out a survey. He was happy to, but then the survey kept going, and going, and going: 43 questions in total. The survey itself soured him on his overall experience with the auto club! A customer satisfaction survey that led to dissatisfaction.
Impersonal Surveys - One day boarding an airline, a steward came to his seat, addressed Jim by name and asked him to complete a paper survey. He said he would be happy to, but then he looked at the survey. It asked his name, frequent flier number, what flight he was on, his destination, and so forth - information that the airline knew, but the survey firm didn't. This made a survey seem as tedious as filling out a form.
Inappropriate Surveys - A hospital sent a household a survey asking the patient to rate his recent operation. Tragically, the patient had died in surgery. His widow was not amused to receive the survey. In fact, she talked to a newspaper about the callous way she was treated, creating a public-relations problem for the hospital.
Disconnected Surveys - Jim said that even Gartner didn't always practice what it preached. The firm frequently ran multiple, disconnected survey initiatives, with analysts often unaware of similar surveys being fielded.
This was just one slide of the dozens Jim presented, and he didn't have time to go into detail on how each of these issues can be addressed. Here are some suggestions.
- Shorten Lengthy Surveys - You can address overly long surveys in three ways: relentlessly prune questions (an approach that will prove unpopular with your coworkers); break a long questionnaire into multiple studies, each fielded separately; or randomly show each respondent a short subset of a long questionnaire. If you are going to have a sufficiently large sample size, then you lose little statistical significance by breaking the survey up into bite-sized chunks. Remember, the mathematics of survey research are such that you do not need to ask every single member of your population their opinion; just a random, representative sample will do. The auto club should have chosen three questions they wanted to ask every customer, then had the PDA randomly choose from the forty other questions; if certain questions only made sense together, they could be set up as a block of questions always chosen as a unit. This way, the auto club would have collected all the data it needed, without making its customers wish they were still on the side of the road.
- Personalize Surveys - Many researchers are used to the old days of paper or telephone surveys, where if information was desired, it had to be asked of the respondents directly. With web surveys, and with web services that can connect disparate applications, surveys can now integrate information from customer relationship management systems, HR systems and other key databases. Yes, this takes more work for the airline to set up the integration, but it would enable the customer to spend their time answering the attitudinal questions in detail, rather than wanting to avoid filling out the form by jumping out the window and using their seat as a flotation device.
- Keep Good Lists of Respondents - Every good survey process should have a global unsubscribe database built in. And the business should have processes set up to add entries to that database. Fortunately, few of us are in industries where our customers may die in our care, but - for those who are - it's all the more important to include that risk in their survey-research processes.
- Connect the Surveys - Most likely your organization has empowered you and your coworkers to make the investments you need to be successful. And often that involves using your credit card to purchase a survey. As an alternative, enterprise feedback management systems empower users to create the surveys they need to field in order to be successful in their jobs, while providing some centralized control to make sure customers are treated with similar respect, and not oversurveyed. Deploying an EFM system across your organization can be a daunting project, but if you start with one brand or one division, you can begin to treat your customers with more respect and better leverage the insights you're gaining from them, and then roll out the solution to other divisions over time.
When surveys go bad, make sure to reform them!
Posted by Jeffrey Henning on Fri, Sep 05, 2008
In the same week, I learnt about two businesses providing a new type of consulting service: Evergance and Voice of the Revenue have both established practices that help companies implement enterprise-feedback management solutions. Both start by developing a detailed understanding of your business requirements, then map those requirements against products from EFM vendors such as Vovici to provide you with an objective determination of what system or combination of systems is best for you.
I'm happy to see this, as it represents the continuing maturation of our industry. If you're uncertain of the survey requirements of your many different departments and divisions, hiring such a consultancy can be a great way to develop a detailed assessment. One of our largest and most successful deployments involved a third party consultancy evaluating a client's feedback needs across the organization, then recommending an appropriately scaled solution.
And, of course, we are always here to help as well, as we have our feedback assessment methodology we can apply.
Posted by Jeffrey Henning on Thu, Sep 04, 2008
As I briefly alluded to yesterday, once when I was in London doing face-to-face interviews, when crossing the street, I looked one way, the wrong way, and stepped into oncoming traffic. I nearly got run over by a motorcycle in a motorcade, and jumping back to the sidewalk, I saw a limo go past me bearing Princess Diana to her next appointment.
Well, my last two posts about our new Mobile Survey - PC Edition product have looked only one way at its capabilities, because that's the way I am most familiar with it: an interviewer using the system to ask a respondent questions and record their answers. In fact, that's typically how I think of CAPI (Computer-Assisted Personal Interviewing) systems.
But, looked at from the other direction, Mobile Survey can also be used for self-administered surveys. Here are a few scenarios:
- Intercept surveys - The respondent is intercepted at a retail location or at a resort and invited to take a survey. The facilitator intercepting them will introduce the survey, discuss the incentive, if any, then have the respondent sit down and take this survey. The facilitator will be at hand to answer any questions that might come up during the survey process and will disburse the incentive upon successful completion of the survey. I've seen this approach used successfully for mall intercepts and at amusement parks. For mall intercepts, the kiosks were great for displaying very large videos, which exceed the ability of many consumers to download and view on dial-up connections; this is a great way to test which of several commercials is most effective.
- Field surveys - For sales calls or field-service calls, the prospect or customer is given a laptop or handheld computer to complete a short survey on their satisfaction with the transaction.
- Kiosk surveys for consumers - A kiosk or workstation is set up for participants to complete a short survey. It could be a trade-show satisfaction survey, positioned near the exits from the exhibition area. It could be a retail-satisfaction survey, positioned near the Point of Sale; such kiosks might be deployed for just a few weeks at a location, and then rotated to other stores.
- Kiosk surveys for employees - In many workplaces, few employees have computer access. For factories, big-box retailers, military bases and similar locations, survey kiosks can be temporarily deployed a few weeks each year to collect employee-satisfaction data.
Posted by Jeffrey Henning on Wed, Sep 03, 2008
With Mobile Survey - PC Edition now released, when would I recommend conducting face-to-face interviews rather than web surveys? The following examples are from my personal experience; hopefully, readers will chime in with more.
- Major-account customer satisfaction - This was actually the primary consulting service we launched the company with back in 1993, initially doing major-account studies in the telecom and manufacturing industries. When you have accounts worth hundreds of thousands of dollars (or pounds or Euros), you definitely can benefit from a personal presence in the interview process. Going to the trouble of a face-to-face interview demonstrates the seriousness and importance of the satisfaction survey. It also can lead to a greater rapport between interviewer and interviewee, drawing out better answers. Finally, in a consumer satisfaction survey you just care about satisfaction in aggregate, but in a major-account satisfaction survey you will often write a profile of each individual company being interviewed. This becomes a very valuable planning tool for both vendor and customer and helps strengthen the business relationship.
- Collecting field examples - For a major copier manufacturer, I once traversed the country interviewing managers of internal copy centers, collecting samples of the color plots, print-outs or photographs that they would like to copy with color copiers. So at the end of the project, not only did I have detailed survey results, but I had a portfolio of representative documents showing exactly what prospects would use a color copier for, if they had one. The illustration I'll never forget was one that a U.S. auto manufacturer gave me illustrating foreign vs. domestic auto market share by state: it turns out that the Midwest has the highest relative market share for domestic cars, dropping as you move out to the East and West Coasts. Nowadays, for collecting field examples, you could use digital cameras, snapping images of respondents and their existing products to creatively supplement the survey results.
- Company repositioning - Two computer manufacturers had merged and were seeking to understand how they could reposition the combined company as the industry migrated away from traditional data-center computing. We surveyed a host of prospects and customers about their perceptions of our client and its competitors, helping to identify strengths and weaknesses that could be used as our client repositioned itself into an IT services company. The face-to-face interviews tended to run long and led to detailed, useful responses. Not all the comments were constructive, of course; I'll never forget how one respondent summed up my client: "A disastrous amalgamation of two very good companies."
- Productizing new technology - Too often the actual practical benefits of a new technology are poorly understood. Face-to-face interviewing, following a general discussion guide rather than a rigorously scripted questionnaire, can be a great way to tease out applications that can benefit from the capabilities of a new technology. We often did such research when investigating emerging telecoms technologies. Focus groups can be used for this application, but I prefer the ability to drill-down to detail one-on-one with a respondent.
And the best reason to conduct personal interviews? Great experiences!
Thanks to doing face-to-face interviews, besides having a laptop catch on fire, I was once almost run over by Princess Di and her motorcade, I once had to deliver a 40-year old bottle of Scotch to a respondent as the "incentive" (and he didn't share a glass with me!), and I once had a respondent who only agreed to the interview because he could tell from my accent that I was an American, and he planned on interviewing me in return for an hour about his four-week itinerary to the States!
This being a blog, and not a white paper, it seems like I have the space to tell a rather long, involved story, about a great experience getting to and from a face-to-face interview. If you just came for the tactical advice above, feel free to skip the remainder of this post.
Castle Donington Village
Brad Patton, our chief software architect, speaks of "the perfect moment" of a trip, that sublime epiphany that resonates within the mind and defines and epitomizes the journey forever after. For me, the perfect moment of my personal-interviewing sojourns came one day when I was driving to meet a Communications Manager. I got on the M1 motorway at Luton and started driving north to Nottingham. My directions were to find Castle Donington Village, take the main arterial through the centre to the other side of the village, and there would be the offices of British Midlands Airways.
As I drove, I kept going through hills and valleys. In the valleys it was foggy, and I had to turn my headlights on; I turned them off at each hill, where the sun shone brightly. At last, I found my way to Castle Donington Village, driving past the villagers cottages to and by an ugly power station. Castle Donington is odd mix of the ancient and the modern. Finally, below me and before me lay a castle, the eighteenth-century mansion of a landed aristocrat. (This is really not a castle, just a manor house, but it seemed like a castle to me at the time: turns out the eponymous castle had been torn down so that its stone could be reused.) I drove down to the "castle" and discovered it had a car park. Not only did it have a car park, it had a sign. British Midlands Airways. My appointment was in the castle!
Actually, reception was in the castle, but the receptionist directed me to a recently completed, modern, replete-with-glass-windows office building, built on a hill below the castle and obscured from sight from the car park. That was where my interview was. The interview took an hour and was memorable for two things: the respondent thanked me for taking the time to interview him (!), and he said, "All our business is about is getting bums on aeroplane seats."
Upon returning to my car, I discovered I had left my lights on. I hurried to it and tested the ignition. No joy, as the British say. I went back into the castle and explained my problem to the doorman. "Could I use a phone to call the AA (Automobile Association)?" I asked. (This was before many people had car phones.)
"What kind of car do you 'ave, governor? Is it gear?" Meaning stick-shift or manual, as opposed to automatic.
"Why, yes it is gear," I replied, wondering what that had to do with anything. (Not having driven gear in the States. To this day I can only drive manual with my left hand.)
"Well, hold on then. Let me get Ted, and we'll see if we can't help you on your way."
He found Ted and the two found their way out to my car. They made me put the keys in the ignition. "When I shout, give the key a turn, governor."
The two Englishmen ran behind me, pushing my car. One gave a yell, I turned the key, the engine rolled over, and I soared off into the perfect, sunlit afternoon.
...and so, while they may not always be practical or cost-effective, face-to-face interviews will always be my favorite method for conducting surveys.
Posted by Jeffrey Henning on Tue, Sep 02, 2008
Right before I went on vacation (and stopped blogging for a bit), Vovici announced Mobile Survey - PC Edition, a new computer-assisted personal interviewing solution for EFM Community. With this product, "sales teams can now gather feedback in person as they meet with clients, military personnel in the field can conduct assessments in areas where wireless Internet connections are not possible, and companies can use Mobile Survey - PC Edition in kiosk-type formats where respondents may take a survey at their leisure (in malls, at tradeshows, etc)."
I'm pleased with this new offering for a number of reasons. First, it is a much easier-to-deploy system than our prior solution, speeding up the fielding of offline surveys. Second, I know the development team had been itching to come out with this improvement for a while, but had other priorities to get to first; it was a labor of love for them. Third, our first-ever product was actually a CAPI solution for conducting face-to-face interviews; Mobile Survey - PC Edition represents the latest in a 15-year pedigree of interviewing solutions that stretches from IntelliWriter to SurveySolutions to Mobile Survey for the Palm to EFM Community.
In the late 1980s and early 1990s, I traveled the U.S. and U.K. doing personal interviews on ISDN, virtual private networks, image management, color copying and other topics that seem rather quaint today. Our first product, IntelliWriter, grew out of my direct experience as a professional interviewer, incorporating the functionality I most wanted to see. The power of our latest release almost makes me want to take a road trip and conduct a bunch of interviews again. Almost.
I have a lot of great stories from my time doing face-to-face interviews, but one stands out, and helps illustrate one scenario where a face-to-face survey is warranted over a web or telephone survey. We were hired by a steel manufacturer to survey their largest customers - each of whom represented over a half-million dollars in business. Given the size of each relationship, it made sense to conducted in-person surveys. At some sites, I would interview up to four respondents, each representing different aspects of the relationship (engineering, service, purchasing, etc.). I would use my laptop to ask the appropriate questions, with our software automatically handling skip patterns and fill-ins. On the last interview of a week-long trip, near the end of the questionnaire, as I typed in a respondent's answer, he said, "Is that your laptop on fire?" Indeed, it was. The power adapter was shooting out sparks and flames, and suddenly the laptop shorted out. I unplugged the laptop, and the smoke stopped. I completed the interview from memory, having neglected to print out a copy of the survey instrument. Fortunately, by then I had done the same survey close to 50 times, and had done it about a dozen times that week, so I really did have the remaining questions memorized. But ever since, I have always carried a paper copy of the survey with me, and I advise you to do the same if you use Mobile Survey - PC Edition. And remember, only you can prevent laptop fires.
Posted by Jeffrey Henning on Mon, Sep 01, 2008
As we in the United States celebrate Labor Day, I will rise to the defense of workers everywhere: don't oversurvey them!
About 45% of our customers survey their employees, in a myriad of different ways:
- Annual in-depth employee-satisfaction measurement
- Quarterly snapshot employee-satisfaction measurement
- Benefits satisfaction and prioritization
- Awareness of corporate initiatives
- Sarbanes-Oxley compliance
- Corporate-policy compliance
- Whistle-blower reporting
- Ethics assessments
- Event planning
- Event registration
- Product/service feedback
- Competitive assessments
- IT help-desk satisfaction
- And many more applications
Once an organization adopts a method of surveying its employees, it quickly comes up with ideas for surveys that it never would have considered before: surveys for preferences for the summer picnic, ideas for names for the company newsletter, even lunch orders! All too often, most of these surveys are done of the entire company, because it so easy to do so and it doesn't have any apparent cost.
But it does have an implicit cost: it burns out employees on providing feedback. Response rates begin to drop, and the feedback is more cursory, with fewer in-depth verbatim responses.
The cure is simple but painful at first. Just like Labor Day, give employees a rest. Use an employee community to set up profiles of employees with facts like: the department they work for, their level in the organization, and other attributes you can use to target survey invitations. Then instead of doing censuses - surveys of the entire employee base - either do censuses of specific departments (e.g., invite all sales staff and sales-engineers to complete the assessment of competitors) or do random samples (e.g., for that quarterly snapshot or "take a pulse" survey, use a random sample across the entire organization). This will help preserve the quality of feedback gathered, without belaboring your workforce!
Error sending email
Email sent successfully
|