The Problem with Best Practices
Do they really exist in data analytics?
Background
In May 2024, I wrote an article about the problem with benchmarking, which struck a chord and resonated with many in the data analytics / data science community. The article covered the limited utility of benchmarks, and why organisation shouldn’t waste their time, and their data practitioners’ time, on fixating with the collection and analyses of benchmarks. This week, I want to plumb a similar vein, this time around this notion of “best practices”.
In my line of work as a consultant (I’m a co-founder of a data analytics consulting firm), I often come across requirements in the scope-of-work on getting information around a particular “best practice” in a domain, be it in marketing campaigns, sales management, etc. It’s one of those things that annoys me, not so much because it typically requires doing some reconnaissance work with the client’s competitor to solicit said information, but because the entire notion of “best practice” flies in the face of logic for me.
And so I dedicate my 81st article on why “best practices” are not best practice.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
What Is Best Practice?
What makes something qualify as “best practice”? A good definition of the term “best practice” is that it’s a procedural standard or set of procedural guidelines that leads to reproducible good outcomes if adhered to. I’m paraphrasing here based on various descriptions out there. Two things strike me in this definition. Firstly, the phrases“standard procedure” and “reproducible good outcomes” — it describes a cause-effect or input-output relationship. Secondly, it implies that I am clear about the “good outcome” that I want before I embark on finding the “standard procedure” that would lead to it.
In the world of business, we know that good outcomes are contextual, and even subjective. And that an input-output type methodology is often confounded by the long-chain of activities between input and output, most of which are difficult to perfectly control, unless they are built into a formal semi-automated operating procedure. We also know from strategy principles that business practices aren’t really comparable; business practices are the implementation-end of strategic choice-making to compete differently in the market. So everyone’s business practice would mirror their unique strategy-choice decisions. So if something is truly “best practice”, it would be limited to a very small domain of work.
Working Towards Better Practice
Management and technology consultants often tout their access and insights to “best practices” across clients and industries. Specifically in the world of data analytics, they might suggest that a particular technology deployment on data management & engineering is considered best practice, or they might suggest that a particular modelling technique for a class of problems is the industry best practice. I have found that this is generally hyperbole. When pressed for hard evidence that such-and-such a thing is best practice, it usually comes down to the majority of the practitioners and industry doing it. This is not good evidence to me. In an evolving domain like data analytics, there is a significant amount of crowd-think and peer pressure to follow without thinking. One simply have to see the cacophony on Gen AI to appreciate how reasonable-thinking corporate executives are hurling themselves like lemmings off the proverbial cliff.
During my time as the regional chief analytics officer (CAO) at Citi (Asia Pacific), I saw many of my regional functional peers looking to identify and adopt (i.e. push) “best practices” from our country to another. Personally, I loathe using the term “best practice” in my conversations with the countries. (I was a country CAO prior to my regional appointment, and so I’m biased towards the processes and practices I’ve built to become successful, and so I try hard not to think that I know better.) Instead, I was particularly interested in countries having significantly different solutioning approaches to similar problems. My curiosity was piqued by their differences in problem-framing and solution design, and my contrarian nature was supportive of diversity in general. I like to go against the status quo, and I like those who do the same; and I want to understand what these practitioners see they led to their divergent thinking. To me, that was the essence of my “best practice” learning across the region. I believed there were “better practices” between one country and another; it may not apply for all countries. I was looking for similarity of market context, talent context, information signal context. When contexts are fairly similar, then understanding how one geography is achieving better outcome than another becomes truly useful and transferable.
Conclusion
I honestly don’t subscribe to the notion of “best practices” in the data analytics / data science domain. There may be very specific areas where such best practices exist and thrive, but they are generally the exception rather than the norm. The nature of data analytics / data science as applied to problem-solving is inherently equivocal (i.e. having multiple interpretations), thereby annulling the application of a singular best practice. But better practices do abound, and it’s for us to recognise similarities and transferabilities so that we can tease these practices out.
