Intuition in Data Analytics
Intuition IS analytics.
Background
- “Intuition is defined as the ability of our brains to draw conclusions through unconscious information, without requiring proof or analytical thinking.” — Centraltest.com
- “Data is objective, unbiased information. Intuition is subjective and risky. Learn why data-driven decision-making is the winning option.” — insightly.com
This pretty much sums up the intense bias against intuition from data practitioners. Intuition is the great enemy of data. During the early rise of data analytics and data science, it was popular to bash those business folks who rely on their gut or intuition to make decisions. And yet, when someone says, “Hmm … that finding seems counter-intuitive!”, what they are really saying is that the finding doesn’t track with the default logical way of thinking. This then implies that if it is intuitive, then it’s commonly logical. So, is intuition logical or not?
And so I dedicate my 71st article to making the counter-intuitive argument that intuition is just another aspect of data analytics.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
Intuition, A Many-Splendored Thing
The standard belief model was that intuition was not critically rational and subject to biased outcomes. That using intuition was the exact polar opposite of being data-driven. But I believe there are 2 kinds of intuition that gets confused. There is the intuition that is honed from repeated exposure — a kind of heuristic default computation. And then there is the intuition that is based on making decisions in the presence of weak information signals against a backdrop of significant uncertainty.
I would argue that the first kind of intuition is not very different from the large language models (LLMs). It works on the highest probabilities that have been coaxed out of exposures to many similar situations. Like an LLM, those probabilities are stored and called upon when a situation is “recognised” as similar. It’s retrieving, not really thinking, although we sometimes call this default thinking. This is the kind of intuition that is meant when we say something is “counter-intuitive”.
The other kind of intuition is sometimes labelled “gut feeling”. Something seems off. It typically happens when you are dealing with a situation that is dissimilar to what you’ve been repeatedly exposed to. The collated and distilled information signal presents a logical conclusion. But you have doubts about it because all the dots are not connected, or in some cases, “forcibly” connected. Some seemingly insignificant pieces don’t fit, but they are not insignificant to you, and you can’t quite explain why. You end up making a different decision from that suggested by the majority of information signals.
Counter-Intuitive
Is it counter-intuitive when you learn that an 18-inch pizza has more surface area (and therefore, more to eat!) than two 12-inch pizzas? Chances are, your answer is “yes”. The simple reason is that 18 is smaller than 2×12. We are not typically trained on the differences in areas; our mathematical training tends to be linear. Therefore, the non-computational retrieval process results in default thinking that the 18-inch should provide less value as two 12-inch pizzas. The failure here is that our brains think we have encountered a “similar” situation that we’ve been exposed to, but in reality, it is not similar at all. This is analogous to LLMs hallucinating.
In diagnostic analytics (see earlier article I wrote on this topic), the knowledge creation opportunity is finding an insight that is counter-intuitive. A scenario that appears “similar”, but is in fact, not. This discovery then forces us to re-evaluate our assumptions and interpretations of the information signals that initially led us to the conclusion of “similarity”. New knowledge is generated that augments the current set of decisions.
That Gut Feeling
The second kind of intuition is deeply fascinating, and complementary to data analytics. In fact, I would argue that it is a form of data analytics. There is a body of evidence that suggests that this second kind of intuition results in remarkably good decision outcomes. It’s typically practiced by those with strong domain experience. This may be analogous to the case of LLMs versus small language models which have been trained on a specific domain or discipline.
Rather than discounting such “gut feeling” decisions as luck, organisations should make the effort to study this phenomenon within their domain context. This can lead to the inclusion of previously omitted information signals in diagnostic and solutioning work.
Conclusion
Data analysts and data scientists need to be able to distinguish between the first or second type of intuition. This is essential for them to effectively navigate the complex decisioning landscape of their organisations. Intuition isn’t the enemy; it must co-exist with data science.
