Welcome to the Listening Post!

Constant Sum & Allocation Questions: Best Practices

Tweet

 

Allocation or constant-sum questions enable the survey author to capture more detail about behavior and are also used to ask respondents to weigh competing priorities.

Sample questions:

• How many visits did you make in the past month to each of the following grocery stores?
• What percentage of your entertainment budget is spent on each of the following categories?
• For the total number of hours of television you watched last week, how many hours were spent on each type of program?
• How many children do you have in each age range?
• When thinking about purchasing a new digital camera, how much weight do you give to each of the following features? (Please distribute 100 points across these items.)

The survey author can force the sum to fit a constant (typically 100% or 100) or to equal a number entered earlier by the respondent (e.g., the number of hours or dollars spent). Alternatively, they can let the respondent enter any amount (in which case this question type is known generically as an allocation question).

Most do-it-yourself survey researchers simply use allocation questions for their form-tallying function, but most of the research literature is devoted to the question’s value for forcing respondents to trade off competing items. In fact, before online surveys, face-to-face interviewers would hand the respondent 10 poker chips and ask them to stack them on different categories to determine their order of importance. Adding a chip to one category was in fact denying it from another, trading off one choice for another.

The constant-sum question, like every question type, has its own strengths and weaknesses.

Pros

• Works well across cultures (no cultural differences in response style of the type that confound crosscultural comparison of ratings questions)
• Allows respondents to assign 0 importance to items (which isn’t possible in ranking)
• Provides ratio data, allowing you to compare the relative importance of items; unlike ranking questions, reveals the relative difference between items
• Respondents almost always differentiate items from another (while it is possible to divide the total evenly between items, respondents almost never do this)

Cons

• Respondents have difficulty making the total reach the desired sum (Cohen & Orme, 2004)
• Takes more time and more mental energy from respondents
• Requires respondents familiar with the question type or with higher education levels
• May not have greater criterion validity than rating scales (Szoc, Thomas & Barlas, 2010)
• When administered on paper often produces data that doesn’t add up to the total; hard to administer on the phone

Best Practices

• Limit the number of items; different authors argue for limits of 5, 7 or 10 items
• If you do have too many items, break them into categories for the overall constant-sum question and then ask subsequent follow-on allocation questions to prioritize subitems within each category
• If appropriate, include an Other category as a safety valve for respondents
• Use matrix questions instead if respondents cannot easily recollect and quantify their past behavior
• Provide visual feedback to the current working sum

According to one study (Conrad, Couper, Tourangeau and Galesic; 2005), providing a running total to respondents offers a number of benefits as opposed to showing a total only when the respondent clicks the Next button. Providing a running total improved the likelihood that the final tally matched the sum desired, it led to more edits of the individual items (implying more consideration of the items) and, surprisingly, it took respondents less time. In a later study (Conrad, Couper, Tourangeau, Galesic and Yan; 2009), the authors showed that providing visual feedback also increased the accuracy of responses. So showing running totals on allocation questions is a win-win.