The Problem with Benchmarking
Why are we obsessed with comparisons?
Background
One of my major annoyances in consulting work is the frequent client requests for industry or competitor benchmarking. As a data analyst, I never understood the value of it. How do you know that the entities in the benchmark are the correct comparable ones? If you are better than the benchmark, does it mean that you are successful in your business strategies and executions; are you looking for a pat on the back? Do you expect the consultant to preserve your confidentiality (of not sharing your information) but violate it for everyone else?
Obsessing over benchmarks is a common malady for many organisations. They think if they have information on market share or industry customer retention rates, that it would lead them to manage their businesses better. I’m not convinced about that. And so I’m dedicating my 37th weekly article to debunking the value of benchmarking.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
What Good is a Benchmark?
When I’m confronted with any piece of data, I always ask the question: “What information signal is being carried on this data?”. There could be more than one information signal. But many times, it’s just noise. And so the same thinking applies to benchmark data. Let’s say you have the customer retention rates of your competitor(s), what can you do with that data? What information signal does it ‘tell’ you?
Let’s start by first defining what exactly is a benchmark? Is a market share report a benchmark? Not necessarily. If the market share report simply shows your organisation’s share against the total market, then it isn’t a benchmark. But what if it shows the market share of your competitors in addition to your own? What if it shows the average of all your competitors against yours? Then yes, that would make it a benchmark report. A benchmark must have a baseline or reference(s) for the purpose of calculating better/worse differences to do comparisons.
Next, what are you intending to do with that comparison information? Now, in my 25th weekly article I wrote about the phenomenon of infonomics and data monetisation. I shared that if you are looking to monetise 3rd party or industry data, then value is accrued towards those data that support decision inputs rather than those supporting evaluatioo of decision outcomes. In the case of organisations desiring to get benchmark information, this is the key question they need to ask themselves: “Are you looking to use that information as input to reduce decisioning uncertainty or are you looking to use the benchmark information to evaluate your decision choices and pat yourselves on the back?”
Types of Benchmarks
You can literally benchmark anything. But in the business world, these benchmarks seem to surface over and over again:
- Pricing benchmarks — capturing comparisons of both input and output price; e.g. product pricing.
- Market benchmarks — capturing comparisons on market share or market penetration; e.g. iPhone market share.
- Satisfaction benchmarks — capturing comparisons on user experiences; e.g. digital app ratings.
- Utility benchmarks — capturing comparisons on perceived value; e.g. product feature rankings.
Each of these classes of benchmarks has their own unique information signal(s). Not all are equally valuable, some are honestly distractive at best. Pricing benchmarks in particular are always useful because they are inherently decision-input oriented. Price carries strong information signals such as quality, availability (scarcity), and even differentiability. If a company manufactures a fantastic product at a low cost, it will still charge a high retail price for it, just so to signal its quality. Price benchmarks fall into 2 categories — publicly available vs internally available. For example, the price for a can of tomato soup is externally available; one can either go online or sample a range of supermarkets to obtain that price. On the other hand, employee salary is considered as internally available. And hence, you have a whole HR consulting industry around collecting salary information through explicit company participation in exchange for getting some aggregated data played back to them.
Market benchmarks, on the other hand, tend to be less useful as decisioning tools in my opinion because of the poor fidelity in their information signals. On the face of it, these market benchmarks come across as serving the purpose of decision evaluation, but I’m prepared to wager that most product folks wouldn’t know how to incorporate them into their product strategies. Market benchmarks suffer from a variety of information signal distortions and mis-presentations. For example, the definition of the market isn’t the addressable market of your business strategy. No organisation competes in the entire market; they create strategies that give them a (perceived) advantage in a particular segment of the market, where deterministic boundaries are near-impossible to define. And so a lower market share may not imply deficiencies in the strategies.
Similarly the inputs on satisfaction benchmarks do not come from persons using all the different competitor solutions and doing a like-for-like comparison. The person liking your product isn’t the same as the person liking your competitor’s product. So why should you care?
Even utility benchmarks aren’t as useful as one would think. Having or not having a particular feature doesn’t make your product better or worse. Each feature is both target market and use context specific. How will we interpret the information signal carried with that benchmark report?
Conclusion
As school kids, we were told not to compare with our peers, but rather to focus on our own internal progress. It’s good advice, but not practical; no one follows it. We are designed to measure progress through external signals. Our internal memories and yardsticks are often biased. And so in our professional lives, we continue this tradition of comparing with our peers. But the problem is, you are not even in the same race with your business peers. The objective of good business strategy is to differentiate ourselves, and to be able to signal that differentiation. Pursuing benchmark comparisons is both reductive and distractive to that differentiation strategy.
We should never start with comparisons unless they help us shape our decision inputs; most don’t. We should ask ourselves what actions we will take or can take from any benchmark report. The quintessential ‘so what’ question. What is the gap in my current knowledge that is preventing me from making better decisions? Would a benchmark report contain all, if not some, of that information to reduce my knowledge gap? In many instances, it will not.
