Welcome to the Listening Post!

Idea Voting Communities: The Challenge of Prioritizing

Tweet

 

The largest research communities tend to be open sites where participants can vote ideas up and down, with My Starbucks Idea being the most well-known of these. Right now the site boasts over 94,000 ideas that have been submitted.

With that embarrassment of riches, how do you decide which ideas are truly the most important? How do you decide which to investigate further, let alone which to implement?

You would hope your idea community had a robust tool to prioritize the ideas.

In fact, most tools – Salesforce CRM Ideas (the engine behind My Starbucks Idea), IdeaScale and Vovici’s own Community Builder – do a lousy job helping you sort through the haystack to find the ideas that will move the needle.

Voting ideas up and down, in isolation of one another, doesn’t tell us much. Sites typically have one or more views showing popular ideas – this creates a self-fulfilling prophecy, as people see popular ideas and rate them. It’s hard for the unpopular ideas to get noticed, even if they are only unpopular because they are unseen.

Showing recent ideas has its own problems, as recent ideas that duplicate older ideas get seen and voted on, distracting attention from the older ideas.

One improvement on traditional voting is pairwise comparison. Two ideas are randomly chosen and then presented for the participant to select from:

After each vote, two more ideas are randomly displayed (try it!). The above example is implemented by AllOurIdeas.org, a free, open-source voting platform for managing pairwise comparisons (free thanks to grants from Google and the Center for Information Technology Policy at Princeton University). You can easily display it within your existing online community.

When reporting the results, how often an idea is predicted to win against a randomly chosen competitor is shown, as in this example:

One of the advantages of this is that a new idea that has only been shown a few times can quickly rise to the top, though you will definitely want to factor in vote count when analyzing the final results. A 75% score with a sample size of four is much less exciting than a 65% score with a sample of 400.

And sample, or sampling, remains a concern with any of these solutions. An idea community that is open to the public is providing a convenience sample and may or may not be representative of your customers. Use the idea community to identify a broad list of popular ideas, but then test that against a representative—preferably random—sample of your target audience to validate the level of interest. The best method of idea prioritization remains a survey. Don’t be surprised if the priority order returned by survey results is very different from that returned by the community alone.

Tips for using Pairwise Comparison

Some quick tips:

• Moderate submitted ideas, primarily to eliminate duplicates. You can see from the above example of results that four ideas are basically synonymous: “improving public education”, “improve the quality of public education”, “improve our schools” and “education”. In pairwise comparison, how would you answer the question, “What is our most important national priority? Improving public education OR improve the quality of public education”. In its defense, All Our Ideas will let you say you can’t decide and then indicate that you think both ideas are the same. But it doesn’t seem to factor this into the analysis yet. Human moderation removes this awkwardness.
• Take great care in phrasing your question. The default example - “What is our most important national priority?” – has two problems. First, it presents a false dichotomy – my initial reaction was that neither was most important; better wording would be “Which of these is the more important national priority?” Second, the current question makes a false assumption about visitors to the site – a common problem with idea communities; in this case, what nation are they talking about? Visitors come to the site from around the world. Make sure you vet your question carefully before publishing it.
• Don’t confuse votes with respondents. With pairwise comparison, if you have 100 ideas for people to vote on, that’s 9,900 possible questions to ask people (100 x 99). One person could vast 9,900 votes.

For a quite different take, see Four tips for making a successful idea marketplace from AllOurIdeas.

At the end of the day, you are collecting thousands of ideas to act on them. Develop a prioritization plan that makes sense for your organization. You don’t want to repeat the mistake of the new UK government: it announced a crowdsourcing initiative, collected 9,500 online comments, and made 0 changes to policy.

You created your community to improve: roll up your sleeves, sort through the suggestions, and get to it!