Welcome to the Listening Post!

Lessons from Polls and the Elections

Tweet

 

As a survey wonk, I love the Presidential election cycle.  Every four years a tremendous amount of attention is paid to polling and survey research, as candidates and the media try to understand the horse race for President.   I learn a ton about the latest techniques and challenges, as research techniques are applied to a unique problem.  For the rest of this week, I am going to write a post a day about polling trials and tribulations and some ramifications for traditional research.

Inspiration for this series is last Wednesday's keynote to the AMA 2008 Marketing Research Conference, What Marketing Researchers Can Learn from Polling Conducted for Candidates and the Media.  Humphrey Taylor, chairman for the Harris Poll presented. 

Humphrey's nine key points:

1. Being a good researcher is necessary but not sufficient.
2. Understanding motivation is very difficult - but vital.
3. Forecasting requires both good research and good judgment.
4. Balance theory and empiricism.
5. Balance trade-offs between quality, speed and cost.
6. Understand what qualitative research does well and badly.
7. Run scared.
8. Continuous quality improvement: the need to change and improve methods and application.
9. Research is no substitute for creative thinking.

Political polls are asking people about something that they might do, but haven't done yet:  vote.  In this way, the lessons of political polls apply to surveys about intent:  intent to purchase, price sensitivity (intent to buy at different price points), willingness to recommend, etc.  Humphrey's next points apply equally well to consumers asked to articulate their next planned purchase as it does to citizens asked to predict their vote:

• People do not understand themselves.
• They rationalize their differences.
• They deceive themselves.
• They lie.

A common staple of survey research is the list of attributes that you rank products on.  Yet buyer motivation, like voter motivation, is very complicated (Humphrey lists ten items that affect voter motivation, from party identification to personal media consumption).

I don't think I've ever conducted a study where respondents didn't end up dramatically overstating purchase intent.  People are cooperative with the interviewer (whether in person or online) and overestimate their likelihood to purchase.  Similarly, when polls go awry in predictions, it is often for overestimating the voter participation rate of key demographics.

In fact, that's just one of six factors that can cause polls to make the wrong prediction.  Here are Humphrey's points for polls and my translation of those to survey research in general:

• Differential Turnout - Just like voter blocks sometimes don't go to the polls as predicted, sometimes customer segments don't make it to market in the proportions you predicted (maybe your channels don't align with the targeted segments).
• Differential Response Rates - Sometimes your respondents do not reflect your target population in the right proportions.  Make sure to factor that into your analysis.
• Bad Methodology - Common sampling techniques are probability samples, non-probability but weighted samples, and convenience samples.  Convenience samples (posting a poll or a link to a survey on your home page) should never be extrapolated to represent your target audience; whether or not you should do that with non-probability weighted samples is a matter of argument (the Harris Poll is based on a non-probability sample weighted for key demographics).
• Sampling Error - Sometimes the prediction isn't wrong, but was inside the range of sampling error.  The more important the decisions that are going to made based on the results of your survey, the more important it is to have a larger survey sample, thereby reducing the range of sampling error.
• Measurement Error - "Sampling error" is just the most well known error, receiving the most attention because it is the most easily estimated.  Measurement error is just one type of non-sampling error:  Different scales can produce different results, poorly worded questions can affect measurement, and so on.  The more important the survey, the more attention needs to be paid to every detail.
• Late Swing - Sometimes market conditions change. If a major event has occurred since your survey (for instance, a competitor has dramatically lowered prices or released a heralded new product), you may need to update your research before drawing any conclusions from it.

For a take on this topic from the other side, see What Political Polls Can Learn From Market Research Surveys, an MRA presentation given in June by Kathleen A. Frankovic, director of surveys for CBS News.  AMA's Marketing News Blog has a post on Humphrey's presentation, as well.

Oh, and don't be too hard on that famous Dewey Defeats Truman headline that I used to illustrate this article.  For two reasons.  First, when elections are "too close to call", they are, well, too close to call.  Switch a few thousand votes in Ohio, Illinois and California (differences all too small to see with polls) and Dewey would, in fact, have won.  Second, the reporter for that story was one Arthur Sears Henning.  As the Chicago Tribune put it, "Maloney banked on the track record of Arthur Sears Henning, the paper's longtime Washington correspondent. Henning said Dewey. Henning was rarely wrong." (According to a poll of my relatives, he's no relation, but I could be wrong on that.)