Welcome to the Listening Post!

Encouraging Honesty from Survey Respondents and Community Members

Tweet

 

Yesterday I blogged about how Dan Ariely's research on behavioral economics can illuminate whether to structure a community using social norms or market norms (incentives).  Today, I'll apply two chapters in Predictably Irrational to online communities to develop some best practices to encourage more honest behavior from our community members and survey respondents.  The two chapters are both entitled "The Context of Our Character" (Parts I & II).  Here Dan explains some of the related studies:

Key insights from Dan's research:

• People are inclined to cheat just a little bit. They won't cheat too much, because that would interfere with their impression of themselves as basically honest.
• Increasing compensation for a task doesn't increase the extent of cheating at it; the mere presence of compensation does.
• The more abstract the compensation (e.g., the further removed it is from actual money; tokens or paper tickets are more abstract than cash), the greater the extent of cheating.
• Complete cheating is rare, when cash is on the line. Only 0.2% of participants were "total cheaters" in Dan's experiments involving cash rewards.  However, when token rewards were offered instead, that jumped to 16% of participants!
• Simply reminding participants of ethics eliminated all cheating.  The reminders took the form of asking participants to recall as many of Ten Commandments as possible or signing a simple statement agreeing to an honor code ("I understand that this study falls under the MIT honor system") .

When I wandered the exhibits at AMA MR Boston, I saw booths addressing different types of dishonest panelists, how to identify fraudulent responses and how to ensure panel quality.  Clearly, the industry is concerned about the veracity of online research results.  So let's translate Dan's conclusions to the realm of research communities and survey research:

• You can eliminate most dishonesty simply by prefacing a survey with a simple Yes/No question reminding the respondent of basic ethics.  For instance, "I agree to provide an honest answer to each question that follows."  For open-ended community research, such as online discussions, a simple reminder to be honest is sufficient.
• When offering incentives, immediate cash payments generate less dishonesty than incentive points, frequent flier miles or other more abstract rewards.
• When cash payment is on the line, it's highly unlikely that members are fabricating all their feedback or that respondents are making up every answer to a survey. 

When considering whether to structure your community around social norms or market norms, keep in mind that market norms may in fact encourage some slight cheating.  To counteract that, make sure to remind people to be honest.  But don't assume that communities based on social norms don't suffer from some cheating as well: in such communities, the motivation for cheating is simply to get through a long survey quickly, lying on some answers to finish sooner.

Oh, and if respondents answer "No" to the question "I agree to provide an honest answer to each question that follows," then branch them out of the survey immediately.  At least they were honest about being dishonest!