“Connecting the Dots” in Data Analytics
The art and science of perspectives.
Background
What differentiates a good data analyst from a great data analyst? People often say that a great data analyst has the uncanny ability to “connect the dots”, as though that explanation makes everything clear. What the heck is “connecting the dots”? Is it an over-used term?
And so I dedicate my 87th article to building up a more precise description of what it means to “connect the dots” for a data analyst / data scientist, and how one can go about developing that competency.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
Equivocal Definitions
A brief discussion with senior practitioners in data analytics / data science revealed what I suspected: that we all intuitively think we understand what it means to “connect the dots”, but when asked to explicitly explain it, we get a bunch of differing narratives such as (I’m paraphrasing here):
- “… refers to the ability to perceive relationships between seemingly unrelated pieces of information that leads to a “big picture” perspective …”
- “… bringing together information to discover hypotheses …”
- “… filling the gaps in your (missing) data through proxy or default logic …”
- “… adding corroborating evidence until you reach a natural conclusion …”
Although related, these descriptions are obviously not similar. So it seems, “connecting the dots” means different things to different analysts, and without a consistent and converged definition, it would be challenging to talk about competency development.
All About Framing
I’ve been doing data analytics work since the 1990s, and over the years, I have been deeply fascinated with the concept of framing and perspectives in data analytics. I’ve been told that I “see” patterns where others don’t, which has allowed me to approach data-driven problem-solving in ways that others don’t often consider. And I’m curious why.
This idea of “seeing patterns” matches up with “connecting the dots”. For me, “connecting the dots” is not about finding supporting evidence for a hypothesis. Rather, it is an intentional act of discovery, first and foremost. It is making something obvious when it was not obvious before.
But where does “connecting the dots” sit in the spectrum of analytical activities? Is it a part of Exploratory Analytics? Is it a kind of methodology in Diagnostic Analytics? Or is it part of Solutioning? “Connecting the dots” sits somewhere in the transition of exploratory analytics to diagnostic analytics. The following conditions must be present for the opportunity to “connect the dots”:
- An event or outcome that has deviated from expectations, for which we are trying to figure out some construct of a root cause(s).
- A set of unusual observations or data points have emerged, but it’s unclear how they are might or might not be connected with the said event / outcome.
An analogy to this condition is the whodunnit detective stories, of which I’m a big fan. A murder has occured, and the detective notices a set of unusual incidences pre and post the event. The detective seeks a perspective where every unusual observation is coherently explained as inter-related occurrences. There is typically only one perspective that achieves that.
In business, this translates to the first principle thinking that the same set of inputs and influences cannot lead to different outcomes. Something is going on, and we don’t yet have a full understanding.
Consider this real-world example: employee fraud has occurred, and there must be unusual data points preceding it, for which we need a “perspective” to be able to “see” and “connect” them. Another real-world example: sales output has suddenly plummeted, and we are figuring out why; something in the extensive data that we are measuring must be able to explain it.
Learning the Art
How then, does one go about learning how to “connect the dots”? In the search for unusual or anomalous data points, how does one distinguish information signal from noise? You need to amp up your abilities in the following ways:
- Develop sensitivities to events, observations, and data points.
- Develop base expectations.
- Develop connected thinking.
Develop Sensitivities
Pay attention to your interactions with your colleagues or with the world at large. Pay attention to what people are saying (because they talk about unusual happenings and not BAU). Don’t gloss over reports; we tend to do it when there’s too much information presented; take the time to intentionally read it. When you do these activities repeatedly and consistently, you will start to become sensitive to anomalous information signals.
Develop Base Expectations
In short, develop an understanding for how things should work. Domain experience is definitely useful but not entirely essential. You can pick up a lot of stuff by reading and conversing. Knowing how things work doesn’t mean you know how to affect them, which is the realm of domain expertise. For example, when you look at the distribution of sale performance data, can you tell if the distribution is NOT per expected? In short, you must be prepared to consume a wide range of information, even if you think it’s not immediately useful or relevant.
Develop Connected Thinking
I like to ask myself: “What else has got to be true?” when I observe a deviated event or outcome. It’s a form of first principle thinking. Deviations do not happen in isolation; we operate in a connected system. Train yourself to keep asking that question. Train yourself to find the answers.
Conclusion
“Connecting the dots” continues to be one of those cognitive mysteries that we have not fully unpacked and understood. This article merely scratches the surface. To what extent can it be trained, just like critical thinking? While not all of us will be great at it, I genuinely believe we can all get better at it.
