That's So Random: Reflections on Randomness in MR
Posted by Vovici Blog on Mon, Mar 21, 2011
Saturday night I invited some of my friends over to play cards and at one point my five-year old climbed into my lap. I had a sudden flashback to being four or five myself, sitting in my father’s lap as he played penny-ante poker with my mom’s extended family. Playing cards has been an important part of my family and social life ever since.
Card games really showcase our lack of an intuitive understanding of math and randomness. I can’t count how many times someone has said to me, “Wow, did you even shuffle?!” When cards come up in patterns, we have a tendency to believe that the shuffling was not random. Yet the chance that you draw two cards off the top of the deck and form a pair is 1 out of 17; drawing three and forming three of kind is 1 out of 425 – not impossible odds, by any means. Moreover, we are mentally programmed to seek out patterns, and we tend to assume that if we see a pattern then the results aren’t random. A good shuffle will still produce patterns.
I’ve seen the concept of randomness confound researchers any number of ways:
- “Hey, I hit refresh on this survey three times and each time the same choice was in the top spot. It must be programmed wrong.” It wasn’t – just an odd bit of luck. (Showing the choices to a question in random order is a great way to neutralize order effects, the tendency of respondents to pick the first choices in a long list.)
- “We randomly routed people down two separate paths of the survey and had 45% complete Part A and 55% complete Part B. What’s wrong?” Nothing. Flipping a coin 100 times doesn’t guarantee you 50 heads and 50 tails. If you need to ensure equal sizes, make sure to use quotas as well as random routing. (Random routing with quotas is a great way to shorten long questionnaires.)
- “We randomly assigned every customer a different quarter of the year to receive our relationship survey, yet the Q3 respondents snap all the trends, but the trends fall back into place for Q4. What’s going on?” In this case, the random assignment happened to lead to Q3-tagged customers having very different firmographics from the other customers of this B2B firm. (Assigning customers to different cohorts is a great way to re-engineer an annual relationship survey, but those cohorts need to be inspected first to make sure they do not have differences that can affect trend reports.)
And researchers confound software developers. Researcher: “I’ll need a systematic random sampling of every 100th panelist.” Developer: “That’s not random at all!” It is, for purposes of sampling, provided the initial panelist is chosen randomly out of the first 100.
And randomness confounds lay people. Again and again I am confronted by people who assume that a large convenience sample is better than a small random sample. Yet convenience samples can produce highly skewed results. While we may not intuitively understand it, randomness is key to the ability to make certain inferences about the populations we’re studying through surveys.
But studying randomness doesn’t all have to be dry. If you attend AAPOR’s 66th Annual Conference this year, please join me at the Applied Probability session Saturday night. The session is a decades-only tradition, and happens to involve playing poker! Card games, besides being a great way to socialize, can help us understand randomness a little better.
See also: