Complexity vs Complications in Data Analytics
That age-old million-dollar question.
Complexity and Complication are NOT synonyms, despite what the dictionaries might suggest. In the world of data analytics / data science, these terms represent very different classes of problems. Many data scientists don’t realise the differences, and hence toil away at incomplete, or worse still, irrelevant solutions.
And so I dedicate my 88th article to why it matters to be able to distinguish the difference between Complexity and Complication in knowledge work.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
Definitions Matter
This is how the dictionary defines Complexity and Complication. They are treated largely as synonyms:
- Complex — not easy to understand or explain; a whole made up of complicated or interrelated parts.
- Complicated — hard to understand, explain or deal with; having many parts that are organised in a way that may be difficult to understand.
But in the practice of data analytics / data science, these words are fundamentally different. In this practice domain, when something is said to be “complicated”, it implies that it is loaded with computational uncertainties. There may be uncertainties in the estimates due to the underlying data and/or underlying computational techniques. For example, predicting the right set of content recommendations (e.g. Netflix) would be considered a complicated problem.
And for something to be “complex” in data analytics / data science, it needs to have two properties: there is uncertainty about how the underlying components or entities are related, and it possesses an emergent quality where the whole is more than the sum of the components. For example, climate is complex; the relationships of the underlying climate components are non-linear and multi-related, making it hard to unpack and simulate. And climate, as a whole, exhibits various emergent properties, like the El Niño and La Niña phenomenons.
To summarise the correct definitions, Complication is equated with uncertainties associated with computations, while Complexity is equated with uncertainties associated with systems behaviour.
Know Thy Problem
It goes without saying that knowing the nature of a problem is critical to solving it. Sadly, the nature of complexity and complication is never really taught to data practitioners. Too often, they apply reductionist thinking to complex problems. That is, they apply a known framework that simplifies the problem into procedural steps, translating it into a complicated problem. The resulting solutions are often incomplete or irrelevant.
Consider the classic example of pricing problems such as optimising the price of your products and services through the econometric concepts of dynamic pricing, price sensitivity, and price elasticity. Pricing is a complex problem. Price can affect demand and supply in non-direct and non-linear ways. Price carries multiple information signals from quality, utility, availability / scarcity, convenience, etc. There is also the network effect of how those information signals permeates throughout the target audience, and the potential distortion on how they eventually get interpreted. Many of the solutions that exist out there employ a reductionist framework, with the solutions skewed towards “optimisation” for a narrow set of stakeholders rather than holistically. Nothing wrong with that if you are aware of that your intended solution is partial. Nothing wrong with that if you are prepared for potential “side effects” consequences.
The ability to therefore recognise whether a problem is complex or complicated allows the problem-solver to enter the fray with the right perspective and make the right trade-offs in the proposed solution.
Navigating Complexity
How then should one navigate and operate against the backdrop of complexities and complications? Data analysts / data scientists are naturally trained to solve complicated problems. To work on complex problems, they need to develop incremental competencies. A critical competency is Systems Thinking (ST). While the field of ST is vast, it intersects with many domains and disciplines. To be clear, data analysts / data scientists are not expected to solve systems problems. Nonetheless, they will encounter them time and time again, and therefore, having situational awareness of such complex problems is necessary.
While ST has a broad collection of cognitive and technical methodologies, the two that may be particularly useful for data analysts / data scientists are (i) System Dynamics (SD), and (ii) Patterns of Strategy (PoS). SD applies an approach to visualise the dynamic interrelationships of the underlying components of a system to reveal (positive and negative reinforcing) feedback loops and non-linear effects. SD is used in multi-stakeholder situations, e.g. climate change, economic restructuring, governance enhancements.
PoS is about dynamic real-time continuous adjustments. It is used particularly in the development and adaptive execution of strategy by seeing it through the lens of actions and opportunities. PoS is used in problems involving markets, competitors, partners, and regulators by continuously assessing their power interplay and fit within the ecosystem.
Conclusion
As the knowledge economy expands, what is becoming evident is its evolution towards ecosystems. This inevitably brings a dimension of complexity into any problem articulation and solution design. It is therefore necessary that data analysts / data scientists equip themselves appropriately. Here, I would like to conclude by drawing your attention to a relevant topic to reinforce. Consider the excitement and noise around Generative AI (i.e. large language models, large reasoning models, agentic AI, etc.). Language is essentially a complex problem, and we’ve used a reductionist framework (tokens and attention) to solve for it as a complicated problem. The root problem is never fully solved or resolved, and this gives rise to unintended consequences. Having an appreciation of this perspective allows us to better navigate the impending AI revolution.
