Designing rating scales
Have you ever thought about how you've designed rating scales? Do you need to rate a bunch of items to identify the most important? Why using an even linear scale is a good rule.
Hi! It’s been about 2 months since my last publishing. I’ve been busy editing and getting my book, “Elements of Product Management” ready for publishing.
This past week, I hit a milestone of getting my first proof print mailed to me for review. Exciting! Secondly, people have asked if I’m shipping this outside the U.S. Before, this was too expensive, but good news! I’ve found a way to ship to the UK, Canada, and India for only $8.99. Now, if only copy editing was this easy.
Recently, I stumbled across the problem of designing ranking scales. Here’s an example of a scale that you’ve probably come across, an 11 point linear numerical scale for an NPS question.
Designing scales is a common technique when product managers need to gather information quickly and cheaply to filter results. For example, you might have a list of opportunities, problem statements, or risks and need to identify which ones to prioritize. You might want the opinions of a few individuals. You see this technique used in the RICE prioritization framework.
Common choices are 3, 5, or 7 points linear scales.
They are easy to create and easy to answer. But as you move up or down the scale, you have to trade-off between:
Precision of data
Ease of use (by the person answering)
As you can imagine, a 3 point linear scale (e.g., rank on 1 to 3 with 3 being highest) is easiest to implement and use. However, if you have many items to rate (more than 10), you’ll lack precision in the data you collect because you might have multiple items all with the same rating. This might require a second attempt at re-rating all the items in a bucket (e.g., all items rated as 3).
This is what is meant by “precision of data”. At the same time, creating a longer scale doesn’t automatically solve this problem because respondents might treat it as a 3 point scale by ignoring the options in the middle.
Thus, I think in most cases where product managers are collecting data with a small group of people (e.g., 5 - 7), working in person or via video chat, it’s best to use an even, 4 or 10 point linear scale.
Even scales forces people to not “sit on the fence”. This increases the precision slightly as you can see in the example above. You can start with a 4 point even scale, which is still very easy to use. Then, once you’ve identified a group, you can move into a 10 point linear scale (1 - 10) for only a subset of the data to increase data precision.
Thanks to the Product Studio students at Cornell Tech who identified this issue and made this article possible. If you have a different technique, drop a note.
Sources