Playing The Devil’s Advocate With Data
How data analytics practitioners can push back to help organisations move forward.
Background
In January 2024, I wrote an article entitled “Seat at the Table (Failure of Analytics Leadership)” that drew attention to 4 important behaviours that data analytics seniors need to get better at if they want to be excel at being partners with their stakeholders. Since writing that article, I’ve actually delivered a successful leadership development programme for data analytics seniors anchored on those same constructs. As I prepare to coach a new batch of data analytics seniors, I thought I would revisit that article, and in particular, unpack one of the 4 behaviours, namely Devil’s Advocate.
And so I dedicate my 79th article on how data analytics / data science seniors and practitioners can up their game when it comes to challenging the status quo.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
Inherently Contrarian
The practice and discipline of data analytics is inherently contrarian. If it wasn’t, there would be no need for it., and we would still be relying on human-based decisioning such as intuition and heuristics. Not to say that human-based decisioning is bad, but it’s definitely far from perfect, and it’s the counter-intuitive or counter-factual insights derived from data analytics that has cemented its role as a winning support and complementary system in organisation decision-making and business optimisation.
So arguably, a data analytics practitioner, a good one, should be naturally contrarian-leaning. (Anyone who has worked with me, or follows my writing, knows that I’m very much a contrarian thinker.) But in reality, too many data analytics practitioners and seniors aren’t doing enough to challenge their stakeholder communities.
Playing the Devil’s Advocate
The term Devil’s Advocate originally derived from the Catholic Church, where an appointee would argue against the proposed canonisation of a person for sainthood by uncovering character flaws. A recent example of that was the late Christopher Hitchens, a world-renowned journalist and humanitarian, playing that role (officially) to speak up against the canonisation of Mother Teresa, in which he presented evidence that was contrary to the prevailing world-view. In fact, he wrote a book about it (The Missionary Position). For a less religious slant, the classic 1957 movie 12 Angry Men is a take on how a juror played the role of devil’s advocate to great effect and outcome.
In common parlance, the term Devil’s Advocate can mean a range of things. It’s been associated with the organisation technique called Red Teaming, a form of decision stress-testing. It is considered a form of critical thinking. It could mean to “push back” against or “challenge” the status quo. Personally, within my data analytics work, I use the term as a shorthand for challenging groupthink.
Devil’s Advocate in Data
Why is the role of devil’s advocate important? At the heart of it, it has to do with the proven value of having diversity in thinking and decisioning. All organisations develop “blind spots” shaped by their successes and failures, failing to account for randomness. In fact, research indicates that subject-matter experts tend to have poorer forecasting outcomes than dedicated amateurs due to various biases and “echo-chamber” affects. Famous experiments include how geopolitics amateurs outperform CIA intelligence experts, and dart-throwing monkeys outperforming professional investment managers when it came to stock picking (this has since been replicated by cats as well!). These days, organisation leaders are drinking the Kool-Aid on the transformative value of generative AI — imagine a recent INSEAD survey shows that 27% of respondents think that AGI (artificial general intelligence) will be achieved within 5–10 years (compared to computer scientists in the field estimating 20–50 years). Everyone is now an armchair expert on Gen AI. Bias and groupthink is everywhere, and the data analytics practitioner needs to be sensitive to it, and to provide a data-backed bulwark against it.
How should a data analytics practitioner play the role of devil’s advocate? I put forward the following: the primary role of a devil’s advocate in data is to improve decision outcomes, not overturn it. It is not an adversarial role but a complementary one. Decision outcomes are improved through the following contrarian-driven processes:
Challenge assumptions and not arguments.
The data analytics practitioner should ask: what are the foundational assumptions that underline your stakeholder’s argument or perspective of a situation? What is the data evidence that supports the validity of these foundational assumptions. Are those data evidence still valid or outdated and even erroneous and mis-represented? For example, as many organisations rush to implement Gen AI, data analytics practitioners should challenge if the choice of large language models used will continue to be available and viable in the next 24 months.
Challenge the “what” and not the “how”.
This idea was contributed by my good friend and data analytics heavyweight Hiok Song Er. The “how” represents the approach, and the experience-based supported premise is that when we challenge the desired (implementation) approach, the recipient becomes defensive, and the challenge begins to feel “personal”. However, if we challenge the “what”, defined as “what are we trying to achieve” or “what will change with the solution approach”, the conversation becomes more expansive and open-ended, and this is where we can probe. Hiok related a story where he was asked to build a real-time dashboard for ATM cash stock by the stakeholders in a bank where he once worked only to realise the bank could not replenish the cash stock in real time. If he had probed on the “what”, he would have developed a forecasting engine rather than a monitoring dashboard.
Challenge the completeness of input data.
The data analytics practitioner is generally the right person to ascertain whether the collection of input data that is being used in the decision-making and deliberation process is complete. For example, have we factored in how competitors might respond to our intended solution, thus changing the market landscape? For example, are there contradicting data findings, and if so, why and what can we learn from it? (A classic case study here is the 1982 new coke launch, which was disastrous and had to be rolled back. 200,000 individual surveys showed a preference for new coke, with about 10% saying they were against the change even if the new formula tasted better. Focus group studies showed that after some discussion, a majority of participants were against the change. This seemingly contradictory data finding highlights the social-network and crowd effects of brands that individual data does not capture.)
Conclusion
Granted that most data practitioners and seniors lack the brassy or charismatic personality to get into a confrontational or challenging conversation with their stakeholders. But courage need not be loud if it’s rooted in clear and objective thinking. In the earlier first article, I stated that data practitioners are in a unique position of having access to a wider range of data and information than their stakeholders, but they need to marry this with the organisation’s strategy and business performance, the market trends and changing consumer behaviours, and connect the dots through their broader interpretation of the data.
