Welcome to the Listening Post!

Favorite Color Survey Results: How Short Choice Lists Lead to Wrong Answers

Tweet

 

Two weeks ago I suggested some best practices for writing choice lists for closed-ended questions, using what I took to be a poorly written "What is your favorite color?" poll as an example of what not to do. Today I wanted to look at how getting a choice list wrong can affect the quality of the results you gather, and how easy it is to get it right.

For background, I researched eight favorite-color polls with different lists of choices.  Each of the eight polls had its own unique choice list, ranging from 6 choices to 15 choices.  Here are the top three choices reported by each poll:

Since each used a convenience sample, none of them can be considered representative of the wider population.  However, I think the results are illustrative of the types of problems that can arise if a choice list is written too hastily.

How close did each poll come to the consensus rankings for favorite color?  Let's look at which color had the greatest frequency, which color had the next highest frequency, and so forth, leaving out "Other":

1. Well, the good news is that all eight reported the same favorite color as #1: Blue.
2. Five of the eight polls all reported the same #2: Green.   Blue and Green were the only colors in the choice lists of all eight polls.
3. The consensus for #3 was Red.  Poll B somehow left Red out as a choice altogether: obviously, one way to get choice lists wrong is to leave out popular choices.  

Polls G & H were the only two polls to get the top three choices right.  What is different about their choices lists compared to the others and what should the lesson be when writing our own choice lists? Simple: Long choice lists are better. Where the other polls had 6 to 10 choices, our two most-accurate polls had 12 and 15 choices respectively.  

Poll H was accurate despite the three flaws that I pointed out last week.  It missed an obvious choice from its list of 15 colors: it lacked the primary color Yellow, which--in a surprise to me--is an unpopular color, with a median rank of 7.  Poll H also had some superfluous choices (no other poll had Hot Pink, Lime Green, Magenta, Maroon Or Sky Blue). And it lacked an "Other".

Poll A is fascinating to me, because its choice list is so logical, including the primary colors Red, Blue and Yellow, and the common colors Black and Green, along with an Other. These are among the most common colors used in board-game pieces, for instance, and it seems as good a set of six choices as any. However, it's missing the popular colors Purple, Pink and Orange, and as a result the "Other" choice was selected more often than the first choice (28% to 25%), a clear sign of a bad selection of choices.  

While only 5 of the 8 polls offered an "Other", none of them actually let a respondent specify what their "Other" was. In your own surveys, make sure to include that "Other (please specify)" so that you can correct your choice list in the future.  

Many of the lists were haphazard but Poll F's list was carefully researched: "The aforementioned colors are listed in Osgood's atlas of affective meaning as colors that carry a high affective value (Cross-Cultural Universals of Affective Meaning)".  

To sum up, the single most important lesson to keep in mind is to have a comprehensive list of choices. Share your survey with coworkers and ask them to help come up with more ideas for choices before you field that survey.  Failure to do so might have you seeing red!

Appendix: Summary of Favorite-Color Polls