4 R’s Mailbag: Retweets, Randomness, Ratings & Response Rates
Posted by Jeffrey Henning on Sun, May 23, 2010
Some questions from the comments section, on counting retweets, the importance of external selection, fully labeled rating scales, NPS and response rates. If you have a question you'd like me to answer, just post a comment to this post.
Retweets Count
Annie Pettit of Conversition Strategies asks, "In terms of retweets [in the context of coding social media comments], think about what they mean to you. Do you blindly retweet? Or do you RT to show that you have a similar opinion? You are borrowing the phrasing of the other person when you RT. It's a big fat ‘This is my opinion too.' So for me, RTs are very important and definitely need to be counted as an opinion."
I personally retweet things that I find interesting, even if I disagree with them. Some people do interpret my tweets (over at @jhenning) as endorsements, which can be awkward. To see if I was alone in this, I asked my followers "Do you agree with everything you retweet?" and five out of the six who answered said they didn't. I think this poses interesting challenges for automated text analytics: its fine for brand monitoring tools to count all these retweets, for instance, since they are measuring the sentiment of word of mouth, but it can interfere with traditional market research. An automated analysis of the reasons for iPad returns would have put "single tasking" near the top of the list, when it fact it was an issue in only one product return (which just happened to be widely retweeted). When, and how, to count retweets and excerpts from blog posts is an important methodological consideration for social media market research.
External Selection
Jen asks, "I'm fuzzy on how doing a random sample is better than inviting all visitors [to take a site intercept survey]. Aren't people going to self-select either way?"
You can't self select in if you're not invited to take the survey at all! A key element of random sampling is external selection--at AAPOR, many of the presentations talked about sending six separate invitations to selected recipients to encourage them to respond. One speaker, in a private conversation with me, talked about sending over 30 requests to participate (which included phone calls for which no voice message was left).
A popup dialog shown to random visitors is an admittedly weak form of selection, but it is stronger than having a passive feedback link visible to everyone.
Rating Scales
Lynne writes, "Jeffrey, nice post [On a Scale of 0 to 10, Numeric Ratings are a 6 Pack of Suckitude]. Definitely food for thought. Two questions: 1. How do you recommend setting up labeled scales in an online environment? My survey software barely allows for end-point labels (readability and presentation concerns), let alone labels for a full 11-point scale. 2. What kind of labels would you use? Most articles I see deal with 5- or 7-point scales only."
Since numbers with labels can confuse respondents, use a standard choose-one or select-one question instead of a numeric scale question type. For writing scales, I use two common patterns, which I describe in my embiggened post Custom Scale Development.
NPS
Gary asks, "Does telling our customers how the NPS scoring is done prior to giving the question, invalidate the score? Because the students now know which scores to pick and which to avoid depending on if they want to support or not? i.e. Explaining how 0-6 are detractors, 7-8 are not used, and that 9-10 are attractors.)"
The difficulty with using NPS for benchmarking is that because it is so easy to use, many organizations adopt it and modify it, yet report their score as if it were consistent. I would say that your instructions will definitely result in you getting very different answers from anyone else using NPS. I would not include those instructions.
Response Rates
Joel Bloom provided a great AAPOR presentation on his work to improve response rates from his student population [see Personalize Email Invitations to Improve Response Rates], with personalization of emails providing the best results. But he still needs creative ways to improve responses further with little expenditure.
One possible suggestion that I did have for you, to improve your response rates. You are asking students for information--why not provide them information in return? Ask them in the survey if they would like to receive a topline summary of the results, and email those to anyone who does. When you send out a new survey, include a link to the results from the last survey. I'd also share results with the student newspaper, where possible. All of that might make surveys seem more worthwhile to students.
Two posts that might give you other ideas:
Got your own question for me? Just post a question to this post. Thanks!