Thinking About Key Performance Indicators (KPI)
The end of December marks Red & White Consulting’s business anniversary, the boutique analytics consulting firm that I co-founded with Sally Taher. I traditionally write an article where I ruminate on the learnings and impressions of the past year.
2021 marks Red & White’s 8th year in operation. Despite the pandemic restrictions, Sally and I have continued to engage in intellectually stimulating work with our clients and we have been lucky to be able to ride out the worst of it. Instead of talking about some of that work as I usually do, I’ve decided to dedicate this 8th anniversary article to an ‘aha’ moment I had from observing the ever-evolving pandemic situation.
I live in Singapore, and as many of you would know, the little country has vascillated between opening up its borders to shutting it down, between allowing more social and dining freedoms and severely restricting it. Those back-and-forths were the (well-intentioned) government’s knee-jerk reactions to incremental information about virus spread, vaccination rates and new virus variants. Despite the government acknowledging that the situation is now considered an endemic, their responses didn’t seem to match their rhetoric — there was no clear boundary definition of what it meant to switch from a pandemic to endemic perspective.
These observations got me thinking about metrics and, in particular, KPI (key performance indicators). KPI are a natural fit in the Decision Science toolkit. But sadly, they receive scant attention and conversation. Instead, too many are fixated on dashboards, getting caught up with the presentation layer rather than the representation efficacy of the content. KPI serve 3 important roles in decision making:
1. Monitor against expectations
2. Define and communicate operating boundaries
3. Modify and align behaviours
Let’s unpack each of them.
Monitor Against Expectations
This is the traditional view of what KPI are … making sure we are working towards meeting our desired outputs and outcomes; it is how a business defines its achievements and successes. Deviations from expectations lead to intervention actions, sometimes prematurely, sometimes too late. Taking a decision science perspective, I argue that more care needs to be taken when crafting these expectation-oriented KPI. We need to consider and test the tolerance or variance levels of these KPI in relation to the activities and outputs of business operations. We need clear definitions of what constitutes a deviation (from expectations) so as to separate noise (i.e. natural variance) from signal (warning signs!). We don’t do enough of this work today. We intuitively expect these KPI to accurately represent the outcomes achieved through a direct line of connected activities and outputs, but oftentimes that is simply not the case as outcomes are confounded by the external environment.
Define & Communicate Operating Boundaries
The change from pandemic to endemic state clearly encapsulates this point about KPI defining and communicating operating boundaries. How should the publicly-communicated KPI defer between the two states? How would the different KPI reflect the different risk considerations and intervention approaches? The strict definition of an endemic is that the disease is constantly present in the community (e.g. influenza) and we live with it; we can’t eradicate it. We monitor endemics for potential outbreaks so that we can ring-fence it, but we don’t live in a constantly heightened sense of alert. It is therefore imperative that the government put in sufficient consideration on what KPI to communicate so that general population can react in accordance with them as rational adults.
In a pandemic, the publicly-communicated KPI would logically be infection rates (or confirmed cases), symptomatic rates, death rates, vaccination rates, hospital capacity. These KPI reflect the theme of threat & response management which is reactive. In reality, infection rates (which has knock-on impact to symptomatic and death rates) were unreliable and understated, making the ability to respond less than desirable. In contrast, in an endemic, I would argue that the KPI should reflect the theme of risk management which is proactive. This could include KPI such as hospitalisation rates of at-risk segments of the population, % unvaccinated of at-risk segments, geographic concentration of unvaccinated population, average crowd sizes at specific public areas.
Modify & Align Behaviours
There is a famous saying “What gets measured gets done.” This reflect the powerful effects of KPI to modify and align behaviours. The trick is obviously in crafting the right kinds of KPI that are linked to stakeholder’s agency, i.e. what is within the stakeholder’s behaviour domain. In business, co-ownership of KPI is common, but can also backfire when the ‘behaviour domains’ don’t sufficiently overlap, leading to failures being spread around through a game of finger-pointing. During the covid pandemic phase, to align various stakeholder behaviours, I submit that it would have been useful if governments would publish the following KPI:
1. Probability of covid infection
2. Probability of symptoms given infection
3. Probability of hospitalisation given symptoms
4. Probability of ICU given hospitalisation
5. Probability of death given ICU
These KPI would be computed for each appropriate age-group band and updated on a rolling 30-day basis. Firstly, it should be obvious to the reader that the multiplication of these 5 KPI would give the probability of death by covid in the age-band population. Secondly, different stakeholders can modify their behaviours in response to a rise and fall in the various probabilities and it makes accountability much more transparent. For example, the appropriate age-band population can choose to restrict their social behaviours (public and inner circle) if the 1st and 2nd KPI increase. Hospitals can increase dedicated capacity if the 3rd and 4th KPI increase. And governments can decide how much resource to allocate to intensive treatment given the rise and fall in the 5th KPI.
Conclusion
In the drive to increase data literacy, organisations often get caught up with the new and shiny such as interactive dashboards and predictive analytics. A solid staple like KPI is often overlooked — it can be a powerful tool for improving decision making if utilised in the right way. Organisations should frequently review the objectives of their crafted KPI — is it to Monitor, to Define Boundaries, or to Nudge Behaviours? KPI mastery will be essential for a matured decision science capability.
