Random Thoughts on 2008 MRA Annual Conference

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

This post wraps up my coverage of the Marketing Research Association's 2008 Annual Conference with some random thoughts and some thoughts on randomness.  My past MRA posts:

• Quality - What's Love Got To Do With It?
• Online Community Culture As It Relates to Market Research
• What Political Polls Can Learn From Market Research Surveys
• What's the Catch? Does Sample Sourcing Matter
• Are You Prepared for the Future of Market Research?

For me, the common themes I heard across the sessions I attended were concern about data quality, concerns about third-party Internet panels in particular and interest - and trepidation - about online communities.

The concerns about data quality were many, and for this I do blame the Internet.  The rise of inexpensive survey software, an application category I pioneered, has certainly contributed to an erosion of the perceived value of surveys.  Because web surveys can be conducted so inexpensively, many end users are not willing to spend the amount required for the highest level of data quality.  If this were an informed tradeoff on the part of research users ("yes, the decisions that will be made from this study aren't that important, so let's not spend a lot" or "the decisions made from this research are critical to the future of the organization and therefore worth spending extra time and money to get right"), that would be acceptable, but as one attendee joked in the Future of MR session, "the one thing you never hear from a customer is 'Take your time'." 

Besides the Internet placing price pressure on all forms of survey research, it has also created an expectation that research can and should be conducted rapidly.  Robin Pearl of Estée Lauder related a humorous anecdote in her keynote.  A salesperson for a panel company excitedly told her how research would be turned around in hours in the middle of the day.  To which Robin replied, "Well, I guess I won't be using your company then."  The salesperson didn't understand that conducting an entire survey in a few hours would not be representative of the U.S. population, who are not yet chained to their email 24 hours a day but have usage that varies by hour of day and by time zone.

I was not surprised at the backlash against third-party Internet panels, because the panel providers have been proactively working to address many of the research industry's concerns about panel quality.  I was surprised at the faith in random-digit dialing.  I do believe there was this Golden Age from the 1950s to the 1990s where randomly composing a phone number and calling it provided a true probability sample.  But I do not believe that RDD today provides a random sample of the U.S. population.  With its exclusion of cell phones, it now underrepresents the youngest adults and overrepresents the oldest adults, and it underrepresents early adopters of technology and overrepresents laggards (as DMS Research showed in their presentation).

That said, without probability sampling you can no longer estimate sampling error, and you can no longer assume that your sample accurately reflects the target population.  The willingness of many Internet researchers to completely ignore this and to treat convenience samples as representative does the industry a disservice. 

But I did not hear a single speaker reference the fact that for an increasing number of businesses Internet surveys can in fact provide random samples!  More and more businesses are moving online:  a favorite bookstore of mine in Boston now reluctantly sells only through the Web, and many other ma-and-pa shops have moved from Main Street to a Uniform Resource Locator.  All such companies maintain email databases of their past and present customers, or - for those who provide Software as a Service - their past and present users.  For them, a random sample is an email campaign away.  They can even schedule a series of reminder invitation emails to reduce nonresponse bias.  They get the benefit of rapid and inexpensive Internet research with the statistical foundation that probability sampling provides.

That meets their quantitative needs.  For their qualitative needs, online communities are the next major trend in Internet research.  Whether end-user organizations build their own community using software such as Vovici EFM Community Builder or partner with a research firm such as Intrepid Consultants, organizations can gather more qualitative information more rapidly than ever before.  The trend is new enough that we can't even decide on what to call this tool - I heard the terms online portal, private online community and advisory council used.  Whatever we call it, it presents a new research resource that can supplement and sometimes even replace traditional research that is becoming less effective.

So that's my take on this year's MRA Annual Conference.  Next up- November 3-5th, MRA's 2008 Fall Conference in Las Vegas, where those of you who really love randomness can bet on it.

Father of EFM

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

On Father's Day, it seems appropriate that I should tell a story about my dad, especially because it gives me a chance to set the record straight. A new customer of ours recently told me that when he received a demo from one of our competitors, the sales person had told him that they had coined the term EFM to describe the market. I had to laugh out loud. For you see, my dad coined the acronym EFM, meaning "enterprise feedback management".

Carl Henning is now a technology evangelist for PROFIBUS field-bus technology, and is a noted field-bus blogger, but in early 2004 he had joined our company to help us grow our international business. He had most recently been the managing director of a Canadian software company. We were thrilled to get the chance to leverage his enterprise-software experience, as we were in the middle of transforming Perseus Development Corp. into an enterprise vendor. We were adapting our flagship product to the Web, and I had been charged with coming up with a name for it.

"Enterprise Feedback Management" mentions in the news

I looked at the major enterprise vendors, the companies we were emulating, and saw that they gave their products glamorous names like SAS/EIS, Oracle OLAP and PeopleSoft SCM. The only problem was that we didn't have a TLA (Three-Letter Acronym) for our industry! That's when it hit me that we were in fact not just creating a new product but we were pioneering a new application category.

We hadn't approached the problem as one of porting our client-server software to the Web, of simply moving from on-premise to on-demand software. Based on extensive customer feedback, we were transforming the product to take advantage of the capabilities the Web offered. Our customers were telling us that they needed to centrally manage survey research, which was scattered throughout the organization. They told us that they needed to set up permissions and workflow so that people throughout their enterprise could participate in the research process as appropriate, some writing questionnaires, some sending out invitations, some only looking at reports and dashboards. Without central management, key customers and key resellers were being oversurveyed, similar surveys were being fielded, and privacy and security policies weren't being followed. Clearly, there were enough emerging needs here for a whole new class of software.

When I said that I wanted to name this industry and then name the product after it, my coworkers thought I was crazy. But I pointed out that where once we had been one of the two first dedicated web-survey software applications (with Raosoft EZSurvey), now there were over 300 survey tools. We wanted to separate ourselves from the pack. I finally won my coworkers over, and we invited submissions from across the company for acronyms. Here are the 46 terms we came up with, including CRM as a point of comparison:

The Google hits and ads were important, because, sadly, there is a finite supply of TLAs. Just 17,576 in fact! And many of them already had common meanings.

My favorite of the ones I submitted was TFM, Total Feedback Management, patterned after Total Quality Management. But all my coworkers loved my dad's suggestion of EFM, and so that's what we went with.

It's one thing to coin a phrase, but it is quite another to give it currency.

Once we had the term, we also began pitching it to the handful of our traditional competitors: companies that like us were shifting to the enterprise market. Why? Well, one product doesn't make an industry. "A rising tide lifts all boats." We especially worked on convincing SPSS to use the term EFM, as the only public company in our space at that time, and we were thrilled when they became one of the first to adopt it.

I always laugh when I stumble across a new survey web site, and they claim to be a "market leader". Last time I checked, market leaders were companies that were actually copied by their competitors. Here's a list of competitors who followed our first public use of EFM back on May 25, 2004:

When we introduced Perseus SurveySolutions/EFM (we had decided Perseus/EFM was too bold a departure for us, and we would get there in steps), there were 0 results in Google for "enterprise feedback management". Today, there are at least fourteen vendors using the term EFM to describe their systems, and Google returns 54,600 hits on "enterprise feedback management". So I'd like to point out that, while I may be the father of EFM, my father is the one who named it. Good work, Dad! Happy Father's Day!

Are You Prepared for the Future of Market Research?

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

At last week's MRA Conference, one of the presentations I attended was entitled, "Are You Prepared for the Future of Market Research?"  Here's a brief summary of the discussion.  (Again, all quotations are from my notes.)

Doug User, Ph.D., a senior VP with Widmeyer Research & Polling, spoke about how consumer fragmentation makes it harder to find and understand target audiences.  "Traditional methodologies were designed for a different culture than today.  Traditional methods have become less reliable and raise issues of data quality.  We need new metrics and methods:  video blogs, online portals, emotional measurement, data harvesting, analysis of comments in online forums, and private online communities."

Mary Ann Packo, the CEO of Milward Brown North America, discussed the challenges for marketers, consumers and researchers, and gave her top 10 changes affecting the future of market research.  For marketers, the challenge is building brands in a fast changing world; for consumers, the challenge is deciding between so many options and choices; for researchers, the challenge is - given all this - how to connect marketers and consumers.  Mary Ann's top 10 changes affecting MR were:
1.    Marketing's new accountability
2.    Integrated marketing and media
3.    The digital explosion
4.    Consumers in control
5.    Privacy concerns
6.    Population and audience shifts
7.    Next-generation consumers
8.    Global evolution
9.    Organizational adaptation
10.    Collaboration

C. Frederick John, a senior executive with MasterCard, provided an end user's perspective on the challenges facing MR's future.  He felt that the industry was on a collision course as demands and processes put pressure on researchers:  time pressure and financial pressure, especially, which together reduce data quality.  Acknowleding he was most likely in the minority, he believes the industry has moved to an overreliance on the Internet and that random sampling is not possible through the Internet.  He is concerned that professional respondents may be demographically representative but not attitudinally representative of the target population.  Fred doesn't trust third-party business-to-business panels at all, as most members are recruited from consumer panels.  He is concerned that quality control is lacking at many data collection houses.  He would like to see a return to the roots of the MR industry, with a greater recognition of the tradeoffs of different methodologies and with a recognition of the importance of projectability and RDD. 

After the three main speakers, there was general Q&A.  Following some discussion of behavioral analysis of web-site usage, one attendee said that web analytics were not enough.  She had started with web analytics and had a detailed understanding of what visitors did at her organization's web site but not why they did it.  She felt that in the future web analytics and survey research need to become integrated (which Vovici has done with our WhyClicked product).

Much discussion centered on how the future should not be framed as "Internet vs. CATI" but that studies should use mixed methodologies.  One attendee mentioned that for her company different types of people took a paper survey versus a web survey, and that by combining both her organization developed a much richer view of respondents.  Another attendee argued that where in 2005 respondents of online surveys were still early adopters compared to the U.S. population as a whole, online respondents today are the mainstream, and telephone studies no longer represent the mainstream but the laggards (as the DMS research indicates).  As a result, for his firm, the laggards - the phone respondents - are not of interest to his technology clients. 

Telephone sample is less representative than it was in the past, many stated.  Because federal law prohibits the automatic dialing of cell phones, many CATI surveys exclude cell phone users; as a result, CATI surveys typically undersample college students and 18- through 24-year old adults.

Are you prepared for the future of market research?  On Monday, I'll address this in more detail and wrap up my series of blog posts on the Marketing Research Association's Annual Conference by summing up what I've heard and what I think it means for the future.

What's the Catch? Does Sample Sourcing Matter

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

At the MRA Annual Conference last week, Melanie Courtright, a vice president with DMS Research (an AOL subsidiary), presented the results of a research study she designed contrasting the results gathered from telephone surveys, Internet-panel surveys and Internet-river surveys. 

So-called "river surveys" dip into the stream of users visiting thousands of web sites to invite them to take a survey, rather than emailing a pre-built panel of respondents.  DMS's Opinion Place river runs promotions across thousands of sites, invites visitors to take a survey, then asks them an 11-question qualifier which helps route each visitor to a specific survey by their demographics.  Her title, "What's the Catch?" alludes to catching these respondents from the river.  (As she described the Opinion Place river, I had to laugh as I envisioned a catch-and-release program for respondents, where they were released back to the Internet after completing their survey!)

In Phase I of the research, conducted in December and collecting 2,412 responses, phone, panel and river surveys were contrasted.  The results uncovered minor attitudinal differences but uncovered significant differences in technology usage and adoption, as would be expected.  For instance, 69% of telephone respondents had been using the Internet for at least 5 years, compared to 80% of the river respondents; 22% of phone respondents don't have home Internet access, compared to 1% of the river respondents; 21% of phone respondents don't have a cell phone, compared to 9% of the river respondents.

An additional key area of difference was in the amount of survey participation.  Only 4% of phone respondents take surveys weekly, compared to 17% of river respondents and 70% of panel respondents.  The average number of surveys per month was 0.3 surveys per phone respondent, 2.9 surveys per river respondent and 16.6 surveys per panel respondent.  Clearly, for researchers worried about the affect of professional respondents on results, telephone or river surveys are better choices than third-party Internet panels.

A Phase II study of 3,647 responses was conducted in April across seven panels.  It found surprisingly few differences between results derived from specific panels, with the difference again being technology usage, as some panels seem to have more early adopters than others.

For data quality, 21% of panel respondents completed the questionnaire in 7 minutes or less, compared to 5% of river respondents.  DMS found quality problems such as answering a trick question wrong or not answering an open-ended question thoughtfully were far more typical of speeders.  Melanie said, "If there is one step you take to improve the quality of your responses, take a look at the responses from speeders."

When mapping the survey results to standard benchmarks such as percent of the U.S. population that are married, are students, are Sunday newspaper readers, etc., each survey methodology produced results close to the benchmark.  However, each methodology produced very different results on technology usage, again as would be expected given that the methodologies differed by technology.

A pointed question from the audience said that probability sampling was the theoretical basis for the projectability of survey research and asked what the scientific underpinnings were for assuming that Internet research was similarly representative.  Melanie answered that replicability is emerging as the standard instead of randomization and that the results from her research were replicable.

Answering the question posed by her presentation title, Melanie echoes Brad Bortner at Forrester to say that sample source doesn't matter as much as people may think.  Melanie's recommendation was that third-party panels were best for surveying low-incidence populations, phone surveys were best when concerned about the skew from online methodologies, and river surveys were best for reaching less surveyed Internet populations.

What Political Polls Can Learn From Market Research Surveys

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Last Thursday at the MRA Conference, Kathleen A. Frankovic, director of surveys for CBS News - and recent winner of the American Association for Public Opinion Research Lifetime Achievement Award - gave a wonderful talk entitled "Market Research & the Presidential Election: Lessons from 2008". She joked that she had accepted the invitation to speak only after saying to herself, "June? Well, the primaries will have been resolved for three months by then."

Political polling is very different than marketing research, but Kathleen said that in the wake of the 2004 problems with exit polls, she had decided to experiment more with MR techniques.

For competitive advantage, CBS News began asking questions in the form, "Would you say you like [your preferred candidate] a great deal better than any other, somewhat better or only a little better?" This question in fact revealed weak support for Clinton in Iowa and strong support for Obama; in retrospect, this was a particularly useful question since in the Iowa caucus a voter can change their vote between rounds, if their earlier preference had garnered less than 15% of the total vote. In New Hampshire, this indicated that Clinton had stronger support than Obama, and in retrospect CBS might have qualified its prediction of a NH primary victory for Obama.

CBS News also began asking the question, "Have you ever recommended someone vote for this candidate?" Again, the results of this question help indicate the relative strength of the support for each candidate in Iowa and New Hampshire.

Kathleen also gave several examples of leading questions used in the past to attempt to measure the American electorate's willingness to vote for a female President. A 1937 poll question was phrased: "Would you vote for a woman for President if she were qualified in every other aspect?" (Thirty-three percent of respondents said that they would.) A Virginia Slim poll of women, in 1970, phrased the question this way: "There won't be a woman President of the United States for a long time and that's probably just as well." (Sixty-seven percent of respondents agreed.) Apparently not only were the survey writers sexist, they didn't understand that you don't ask leading questions. Unfortunately, I didn't write down the current wording that CBS uses, but the basic approach is to ask respondents if they would ever vote for a candidate who was a man/woman/black/Hispanic/atheist/etc., if most people they know would ever vote for such a candidate, and if America is ready for such a candidate. (Eighty-one percent said they would vote for a woman candidate, but only 56% said most people they knew would and only 60% said America was ready for a female President.)

For more from Kathleen, check out some of her articles for CBSNews.com:

• How Does Age Affect Polls? Older Respondents May Process Questions Differently
• Why Do Polls Yield Different Results? A Look Inside The Numbers Shows Why Similar Polls Can Seem So Different
• Do VP Choices Matter? CBS News' Kathy Frankovic Breaks Down Polling Evidence From Past Presidential Elections

Online Community Culture As It Relates to Market Research

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Last Thursday the MRA Conference had a great panel discussion on communities and market research.  Some highlights follow. (The quotes are reconstructed from my notes, as no transcript was provided.)

After much discussion of consumer communities, one of the questions from the audience was on how applicable online communities were to the business space.  Misia Tramp, founding partner and president of Intrepid Consultants, said, "Eighty percent of our online Advisory Councils are B2B.  Because relationships in the B2B space have a higher cost to switch than in the B2C space, B2B panelists actually contribute more than consumer panelists do."  She went on, "Think about it - today, surveys are often the only B2B engagement that many business customers have with the brand."  To the issue of incentives, she said, "None of my members of any of our Advisory Councils are incentivized.  We give the counsel a clear manifesto, a clear mission.  Members feel a sense of responsibility to contribute."  Finally, besides B2B, in what she termed "the employeeverse", Intrepid is creating Employee Advisory Councils within organizations.

Speaking of employees, Misia said that traditionally internal market researchers impose themselves between respondents and the business users within their organizations.  Speaking metaphorically of a focus group, she said that stakeholders need to come out from behind the one-way mirror and sit at the table with the respondents.  Intrepid prefers to include stakeholders in the conversations by giving them real-time access to the Advisory Councils.  She believes this provides better insight and greater, more constructive engagement on the part of respondents.  "Online portals make research more scalable and affordable so that you can do more research," she said.

Another question from the audience was on the issue of bias.  Murtaza Hussain, Chief Executive Officer of Peanut Labs, responded, "Every online community or subcommunity has a bias:  people like to hang out with like-minded people.  You have to define the bias, not fight against it."

On the topic of going against a community's culture, Murtaza said, "You can certainly generate a backlash.  If you have a boring, long survey, or you don't fulfill your commitment on incentives in a timely manner, that one bad experience will be multiplied across the community: that one unhappy member gets read thousands of times.  As a result, we don't do certain types of surveys."  Misia concurred, saying, "You now have an accountability to the community."  Surveys need to be attractive and streamlined.  A survey can't have many very similar questions. The content has to be engaging.

As I said, it was a great panel discussion, with a lot of good questions from the audience and good responses from the panel.  Clearly, online communities pose some fresh challenges for market researchers.  Traditionally, little thought was given by market research departments to the respondents who took the survey; they were a disposable commodity, often purchased from the lowest bidder and subjected to long questionnaires.  With online communities, the respondents are now members who need to be engaged and respected, for they now have a forum to air their grievances.  Traditionally, internal users of the research were not involved in the process until the end; now they can be community members participating alongside respondents, teasing out better insights into the constituencies they serve.  While the challenges posed by feedback communities are new, all panelists agreed that the rewards are great.  When it comes to gathering qualitative data, online communities are - as Murtaza put it - "faster, cheaper, better."

Quality - What's Love Got To Do With It?

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

I attended the Marketing Research Association's Annual Conference last week in New York City.  The MRA Conference is a great opportunity to gather with colleagues from across the industry to discuss the state and future of MR.  This year's keynote was particularly inspiring.  Robin Pearl, vice president of Market Research at Estée Lauder, taking her lead from Tina Turner, asked, "What's Love Got To Do With It?" when it comes to achieving high-quality market research.  Robin addressed the quality of sampling, questionnaire content and analysis.

Many issues are affecting sampling.  Neither of the most popular quantitative methodologies - telephone interviews and Internet-panel interviews - now accurately represents the population.  The 18- through 24-year-old demographic frequently uses only cell phones, putting them out of reach of most automated dialers, which only dial land lines.  On the other side, the 55+ demographic is less likely to use the Internet.  In addition, different ethnicities are harder to reach by different methods.  As a result, Estée Lauder has encountered some of the following quality issues:

* Errors in Weighting - One survey reported in error that few women under the age of 25 used lipstick.  This error was a result of the fact that the results had not been weighted to reflect demographics at all and had simply underrepresented that demographic.  Robin also reported occasions where weights were applied to the final results rather than to the screener, providing results that did not accurately reflect that different demographic segments did in fact have different propensities of usage.  When weights were applied, some research firms showed a willingness to weight a sample size of 50 up to 250 for a particular segment; only minor weighting should be applied, to prevent distorting the results.

* Oversurveying - Robin pined for the days when screeners typically excluded respondents who had participated in another survey within three to six months of the current survey.  She joined a number of Internet panels and was shocked to soon be receiving 10 invitations a day.  Professional respondents may in fact differ attitudinally from casual respondents.

* Panel Overlap - Estée Lauder is concerned about professional respondents being double counted.  In one survey Robin sponsored, there was significant overlap of respondents across panels, to the point where through different panels the same household completed the same survey on the same day.

Besides issues with the quality of sampling, Robin has encountered quality issues relating to questionnaire design, especially for face, content and predictive validity.  Too many surveys are fielded without adequate review of the survey instrument and without pre-testing.  Pre-testing is important to determine which questions are unclear and what topics should be covered that aren't.  One of Robin's pet peeves is questions that ask "What do you like about....?" rather than "What, if anything, do you like about...?" (I confess I'm guilty of this.)

Finally, Robin is concerned about issues with data analysis.  She prefers top-box analysis to means, finding the reporting of means to be too reductive.  If she had two products, each rated 3.0 on a 5-point scale, and one product was always rated 3 by respondents (0% on a 2-point top box), and one product was always rated either 1 or 5 (50% on a 2-point top box), she'd go with the "love it or hate it" product.  She also pointed out that too often we as market researchers shortchange the conclusions and recommendations.  Obviously, this work has to come last, requiring an accurate and detailed analysis to proceed it, but as a result we are often hitting against the final deadline and don't provide the insight and implications that we could have if we had budgeted more time.

Robin concluded her MRA keynote by saying that each of us need to have a Ph.D. when it comes to research.  Not an academic degree, but "Passion, Heart and Drive".  Quality research comes when you "love and respect your work", she said.  That's what love's got to do with it.

Her keynote was well received by both end users and researchers alike in the audience.  And while Robin was inspired by Tina Turner, I thought of another diva, Diana Ross: "You Can't Hurry Love."  Many of the issues with quality come from our constant hurrying of the work.  You can't hurry love - or quality.

Opting Out Of Feedback Communities

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Seth asks, "Do you give an option to 'opt out' of the online community? Can they drop out without communication to the administrators?"

For our Vovici Community Builder, members can certainly opt out of receiving emails (newsletters and survey invitations and reminders) simply by clicking an unsubscribe link at the bottom of those messages. They are still members of the community, and can go to the community website and log in whenever they wish, to see messages and surveys they were invited to since they last logged in.

As a result, the community administrator can still invite such unsubscribed members to take a survey, but they are not going to receive an email invitation; the new survey will show up in their list of surveys, which they will only see if they decide to log in again.

Any member can passively opt out, of course, simply by ceasing to visit the community site.

Benchmarking of Interest but Little Used

Email Article |  digg it |  reddit |  delicious |  StumbleUpon |  Facebook |  Twitter |  LinkedIn 

Vovici just completed a survey of 334 North American organizations about their use of benchmarking for customer satisfaction, employee satisfaction and course evaluation.  By far, the greatest interest was in employee satisfaction:  33.6% were interested in benchmarking employee satisfaction and loyalty against that of other organizations.  Despite the healthy interest, only 4.3% of responding organizations actually did such benchmarking.

The sample size of organizations using any type of benchmarking was small.  However, 76.7 percent of such users were very satisfied or somewhat satisfied with the benchmarks their organization currently uses.  For organizations that were neutral or somewhat dissatisfied, this was because the benchmark was incomplete, from their perspective:  respondents desired additional breakdowns (by industry subsectors, by geographic region) and additional detail (not just employee-reported satisfaction levels, but comparable turnover and retention statistics as well).

Surveying was conducted in May, 2008, using a convenience sample and is not representative of North American organizations in general