Don't Ask, Don't Know
Posted by Vovici Blog on Thu, Jan 13, 2011
One of my more controversial tweets in Tuesday's #MRXU Twitterversity event was "Avoid numeric rating scales, label each choice, lead with the negative choice first and omit 'Don't Know'. http://t.co/4Rz9A1Z #MRXU". As my views on this are far more nuanced than I could do justice to in a tweet, this seems like a good time to re-run my post from last May. And do check out the excellent comments.
Arie asks, "I read the post re rationale for not including ‘Don't Know' and
‘No Opinion' options but was wondering if you had any other reasons for leaving these out. Personally I'd rather have some
satisficing rather than forcing respondents to answer questions that they are unable to."
The primary reason for leaving out these options is that by including them you get less data. The secondary reason is that you skew your data: you get fewer results from the less educated and the less motivated (which might be more infrequent customers in a satisfaction study). To compensate for the first, you need larger sample sizes, but the second is harder to compensate for. Best to omit these choices. Remember, according to the paper The Impact of "No Opinion" Response Options on Data Quality, most respondents do have an opinion but your inclusion of a "don't know" choice gives them tacit permission to not have an opinion and to not do the mental work of thinking the question through.
Like you, I would rather have satisficing than wrong answers. In fact, as a respondent, I have quit surveys precisely at the point where an answer was required, and I didn't have the option for "don't know" or "no opinion", even though that was the right and honest answer for me.
My compromise of these two points? Don't show a "don't know" choice, in order to avoid tacitly encouraging such responses, but (in web surveys) don't require answers to questions for which a "don't know" might be the right answer. Don't ask "Don't Know"!