Statistical Subjectivity – The Essence of Rankings
I ran across a good article by Malcom Gladwell in the February 14 & 21 issue of The New Yorker titled, “The Order of Things.” The detail with which he explores rankings of colleges, hospitals and cars demonstrates the immense subjective potential rankings have. What is even more astounding is Gladwell’s discovery of the degree to which many organizations hold their leaders accountable for their place in these rankings.
From an intuitive perspective, people tend to have an emotional connection to statistics; they satisfy feelings for certainty, clarity and knowledgeableness. Thus, when we express arguments statistically, they tend to carry more weight than if we simply express them in words. Rankings clearly define for us what is best, better and good. However, they are more akin to magic where reality is but a trick. Thus, the feelings we receive from rankings (certainty, clarity, knowledgeableness) are satisfied because we want to believe their magic is real.
As a rule, unless the ranking is comparing very similar things against a single, measurable criterion, it is highly subjective. Therefore, here are some important questions to ask about the ranking to discover how its trick works:
- Is it really comparing similar things?
- Is the ranking based upon multiple criteria?
- How important is each criterion and is it valid?
- How does it weight the criteria?
- Is it using some criteria as proxies for things that are difficult to quantify or research?
- What important criteria are absent because of these difficulties?
- Is the difference between one rank and each of those immediately above and below it that significant?
- How accurate was the data collected for each criterion?
- What problems might have retarded data quality?
Applying these questions will demonstrate that our affinity for rankings is more emotional than pragmatic.
About Post Author
My scrum master training introduced me to the idea of using fiibonacci numbers to rank value rather than sequential ones, for the very reason you are describing. The emotional content can be forced into a rigid ordering, which is recognized to be false
Because rankings are emotional, always. It’s baked in. So, if that is a risk, it should be recognized and mitigated in the process
So, give me an example of how Fibonacci numbers would be used in rankings? Instead of 1,2,3,4,… would it be 0,1,1,2,3,5,8,13,…? When you say the emotional content can be forced in a rigid ordering, are you saying that emotionally there would be a bigger difference between 1 and 2 than between 99 and 100 than simply saying both are a difference of 1?
Yes, you’re right. Rankings are always emotional. While it can be mitigated, I don’t see how it can be eliminated with any kind a enumeration. True?
yes, the agile/scrum framework requires a small group of people do rank things. Giving them a set of fibonacci numbers
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377
To rank a set of priorities, first rank them separately, and then as a group explain why they individually came up with their ranking..and then through discussion arrive at the group’s consensus
…they do this for the very reason you describe. Popping people out of the usual sequence makes them have to recognize their intuition about the relative importance of different goals
Thank you, I see now, Murphy. It also helps to do this with words by using words that are less familiar. Sometimes words carry baggage making communications hard because there is little agreement on what a word means. Consequently, people end up talking around one another. For example, I once used “inputs” to describe resources, assets, capital, people, inventory, etc. While cumbersome at first, once they built their own group definition around it, things began to flow. Thank you again for the explanation.