Humility and Data Analysis

Email Article |  Facebook |  Twitter |  LinkedIn 

Over the years, I've read lots of discussions about how to analyze data, with many explanations of statistical methods, yet never once have I read anything that encouraged  analysts to set aside their worldviews.  As we in the United States anticipate tonight's Vice Presidential debate, I can't help but recall last week's Presidential debate, where CNN showed the reaction of Republicans, Democrats and independents as each candidate spoke.  I'm sure to no one's surprise, when Obama spoke, Democrats approved and Republicans disapproved;  when McCain spoke, Republicans approved and Democracts disapproved.

Now comes new science that says that the political party a person belongs to affects how they analyze information, especially how they analyze incorrect information (When Corrections Fail: The persistence of political misperceptions and The Enduring Importance of False Political Beliefs).  Partisans are less likely to believe factual criticisms of their candidate, and more likely to believe unwarranted criticisms of their opponents' candidate. Having the research to back it up is great, as it's something I've always intuitively believed. 

This is one reason why I am a political independent.  Being neither a Republican nor a Democrat has made watching the dialogue at Election.Twitter.com especially enlightening for me.  I've seen lies and half-truths championed by partisans on both sides, yet spoken by people who clearly believe what they are saying. 

How does this relate to survey research?  People-with all their biases-write surveys, and people-with all their biases-analyze surveys.  This presents many challenges for market research.  The world view of the survey author, the survey analyst and the decision maker all need to be kept in mind.

• Survey Author - In questionnaire design, obvious leading questions are easy to spot: "Mayor, which is your preferred method of stifling dissent, banning books or burning them?"   Unintentionally leading questions are harder to spot; sometimes the survey author writes a question using terms he or she considers neutral that aren't neutral.  For instance, "Are you pro-choice?", "Are you pro-life?" or "Do you support abortion rights?" are all leading questions.  Instead, look at how the Harris Poll approaches this sensitive subject:  "The two main groups in the abortion debate are the so-called pro-life group, which opposes abortion, and the so-called pro-choice group, which supports women's right to have an abortion. Which one of these groups do you tend to support more?"
• Survey Analyst - Sometimes, the results to a particular question are so surprising that the analyst can't believe them.  Occasionally, this is for good reason, because the data is wrong:  the question was ambiguous or the choices miscoded or the scales reversed or some other mistake has been made.  Frequently, though, the data is right, but surprising.  Too often, in this case, the analyst will look for reasons to explain away the data rather than embrace the insights provided by this surprise.  From early work I did with Bill Ablondi on mobile professionals, I remember being surprised that professionals with the most interest in the product concept of a PDA were not, as Apple was relentlessly championing, people who did not yet use computers.  Even though Apple was crafting its entire Newton strategy around that market, the greatest interest in fact was from people who already had computers and now wanted access to digital information from their computers in a more portable form.  In hindsight, this is obvious, but it was not obvious at the time, when I was caught up in Apple's reality distortion field (a great phrase for the corrosive effect of world views).
• Decision Maker - Nothing is more disheartening to the analyst then presenting data to the client or decision maker and having it rejected outright.  The client literally refuses to believe the data.  I recall one product-planning project where we took the respondents' reported interest in the solution at any price, multiplied that by purchase likelihood at different prices, and built it into a financial model of likely sales over time.  We were not saying launch or don't launch the product, just reporting our estimate of sales.  The client was furious that this analysis significantly underrepresented the opportunity.  We had verbatim comments to back up the analysis, and stood by it.  The client ignored the advice and built and launched the product anyway.  Its sales didn't even reach our forecast. 

All the best practices of survey research and all the proven methods of statistical analysis are useless if the survey author, analyst or decision maker lets their world view get in the way.

If many Republicans see the world through rose-colored glasses, and many Democrats see the world through blue-colored glasses, tonight, when we watch the debate, we should each try to see it from both angles.  That might help us replace a potentially two-dimensional view of the event with a richer, three-dimensional view.  Because often times the analyst is as important as the analysis.