Welcome to the Listening Post!

Market Research Statistics: Common Pitfalls & Mistakes

At the AMA Applied Research Methods conference last week, Neil Helgeson, a senior methodologist with TNS, taught a workshop "Basic Statistics for Marketing Research". Throughout the session, he highlighted common mistakes that he saw market researchers make.

Misusing Means & Modes

• Don't use means when analyzing rankings. In ranking questions, the item ranked first by a respondent may not be the same distance from the item ranked second as the second is from the item ranked third; report frequencies instead.
• Don't use means when analyzing open-ended numeric fields. High outliers can dramatically increase the mean; use the median instead.
• Don't use the mode (the most frequent score) when there is a large dispersion of answers. In such a case, the mode is "a nonsense value"; as an alternative, report frequencies of distribution.

Correlation Mistakes

• Confusing correlation with causation. Ice cream consumption and street crime may be correlated for a particular Northern city, but ice cream isn't causing street crime [though here I pictured Penguin foiling Batman with some bizarrely flavored cones!]. Correlations are often to other factors, which may or may not have been measured: for instance, for this example the underlying cause is probably hot weather.
• Not recognizing how restricted ranges disguise relationships. Restricted ranges may make a relationship between two variables look weaker: for instance, SAT scores are only loosely correlated to college CPA for Ivy League colleges, because the SAT scores that gain admittance to such schools occupy a narrow band.

Misunderstanding the Role of Sampling

• Acknowledging our use of convenience samples. The theory of statistical inference assumes random sampling but "this is almost never true in marketing research; the less representative of the population our sample is, the less valid our conclusions will be."
• Remembering that populations are more important than samples. "If we acted as though our sample was a perfect representation of the population of interest, we would frequently make decisions that were not optimal."
• Large samples may be overkill. They let you draw conclusions about smaller differences. "If it takes samples of 1000 to see a significant difference between products, there is not a practical difference between them," Neil said. "The importance of making errors decreases as the difference between the null and alternative hypotheses decreases."
• Small samples may be worthless. "We conduct studies with the goal of rejecting the null hypothesis. If we fail to reject, we conclude nothing. Therefore, there is a little point in conducting a study with little chance of rejecting a false null hypothesis. You've spent a lot of time and money to reach no conclusion."

Confusion about "Statistical Significance"

• Samples are not "statistically significant". The phrase statistically significant is often misused; differences between answers are statistically significant, not certain sized samples themselves.
• Statistically significant results may not be meaningful. While a difference may be unlikely to have occurred by chance, it may not be large enough to be important.

Misapplying Significance Tests

• Statistical significance tests are an aid to interpretation, not the interpretation. "There is nothing magical about it. It is a piece of information that tells you how much weight to give to the difference." It provides guidance. Don't equate statistical testing with decision making.
• Using statistical significance tests "for everything against everything". Significance tests are not designed for such use. Test only what you expect to be different.
• Violating the assumptions required for statistical tests to be accurate. Parametric tests in particular tend to make assumptions about the data that may not match the data that you are analyzing. "Typical assumptions include: observations are random and independent, distributions are normal, variances are equal, measurement is at least interval level. The last three are almost always violated to some extent."

As Neil said, "Statistics is one of those things that if you don't use it, you lose it quickly. My goal is to help you be informed consumers of statistics, and we are all consumers of statistics. Some of us outsource to statistical experts, and statistical experts outsource to our statistical software!"