The Karate Kid Weekend Mailbag
Posted by Jeffrey Henning on Sun, Jun 13, 2010
I just took my daughter to see The Karate Kid, and I'm wondering what market research led them to name it that when it is about kung fu and set in China? Nostalgic desire for a remake trumps American cultural illiteracy among U.S. movie consumers? Anyway, their research was right, as it won the weekend box office.
Anyway, time to catch up on the blog mailbag.
Rating Scales
Matt writes, "No one [commenting on Rating Scale Comparison] has mentioned that agree-disagree items are prone to acquiescence bias (though Jeffrey hints at this). Notice also that in the original blog post, Krosnick's response is the only one citing scientific evidence for his recommendation. Read the literature."
You are right, Matt - we should read the literature. Unfortunately, for many researchers, rating scale best practices are a matter of faith rather than science, a matter of tradition rather than measurement.
And to add to your point about acquiescence bias, I would encourage people to rewrite agreement scale questions; that's another tradition that should be retired.
Readers Point You to Their Writing
George Kuhn writes in to suggest his new research blog, The Research Bunker. Check out his post, Data Integrity: Protecting the Quality of Your Online Survey.
Rob Markey suggests his piece in the Harvard Business Review, "Closing the Customer Feedback Loop". I enjoyed his article, as well as one submitted by Saeed Khan, who suggested his Pragmatic Marketing how-to, "A Model for Metrics-Driven Feature Prioritization".
Response Rates
In his article, Khan writes, "Our response rate for paper-based surveys averages 70%, whereas a response rate of 10%-15% for Web-based surveys is considered high."
I don't know anyone who would consider that to be a high response rate, but this is becoming a common meme: "response rates to web surveys are 10% and under" (I heard it again at the MRA conference this week). First, if you have less than a 20% response rate, you have to be seriously concerned about non-response bias (see Optimize rather than Maximize Response Rates). Second, such a low response rate is a sign that you doing many things wrong: see Improving Response Rates to Web Surveys for some suggested best practices.
In a comment to that post, Joy Levin of Allium Research writes, "One incentive we have used for some clients in B2B is information (e.g., white paper, copy of report highlights, etc.). Does anyone have any thoughts on/experience with this type of incentive? We have had success with it, but we've wondered about the experience of others."
Joy, I love them -- as I've written before, "We have long recommended [sharing survey summaries] as an incentive, but I was surprised to see that 94% of e-Rewards panelists valued them [see Online Surveys of Business Decision Makers]. Survey summaries costs little to prepare and can be selectively edited to omit confidential or competitive insights. They are a reciprocal incentive -- you are asking for information from the respondent and providing information in return. Perhaps as a result, such an incentive tacitly encourages honest and candid responses."
Don't Overlook Translating Error Messages
Geraldine comments on the difficulty of multi-country projects:
The biggest challenge is on the actual design of the survey especially when it comes to the many possible error messages that are mostly as a result of Internet related issues - such as ‘cannot find page', ‘time out". Mmmmmh, having to translate such possible errors to the various languages is no joke. The web designer has to think of all likely errors then I have to get these translated. The good thing is that these are more or less standard errors. Nevertheless response from non English speakers is normally very low...and I tend to think it's because of error messages they get that have been overlooked in the translation.
A great point, Geraldine, and one I've seen overlooked when translations are prepared. For more on survey translations, see Survey Translation from 3,000 Feet.
As always, I appreciate your comments and questions. And I still feel cheated by The Karate Kid. It would be like going to see "Planet of the Apes" and having it be about a planet of lemurs.