We Don’t Understand AI At All
Are we just naive, ignorant, or misinformed?
Background
I had recently watched the viral YouTube video of podcaster Dwarkesh Patel interviewing Richard Sutton (winner of the 2025 Turing Award for his contribution to reinforcement learning), and prior to that, I was co-instructing an adult beginners class on Machine Learning. In both these instances, I was struck by the questions being asked about what AI might be capable of doing:
“Why can’t AI detect security flaws in our system autonomously?”
“Aren’t large language models (LLM) already generalising to solve a bunch of Math problems?”
“Can’t I use AI to build machine learning models, from data collection to feature engineering to model selection?”
I’m getting the sense that despite all the information coverage and supposed AI literacy programmes out there, most people don’t really understand what AI is. Is it a case of being naive, ignorant, or misinformed? Citi recently announced that it will train 175,000 employees on how to write better Gen AI prompts. Accenture similarly announced that those who cannot be re-skilled on AI will need to exit the company. These companies are still thinking that AI literacy is equivalent to using pre-built Gen AI solutions. Despite employees knowing how to leverage some ChatGPT prompts, many seem to have quite strange expectations of what AI can actually do.
My 112th article explores this mismatch in expectations and understanding of AI in the corporate world and society.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
What is (Artificial) Intelligence?
For most people, when they hear the term “AI”, it evokes images of animated intelligent robots because so much of this is deeply influenced by literature. We are fascinated by the creation of synthetic or artificial life. From the ancient Greek myth of Talos the giant bronze automaton, to Mary Shelley’s 1818 novel Frankenstein, to Isaac Asimov’s I, Robot series published between 1940 and 1950, to Stanley Kubrick’s film 2001: A Space Odyssey, to the more recent Star Wars, Terminator, and Matrix movies. With each iteration, AI acquires more god-like abilities. All seeing, all knowing, able to accurately predict the future. These descriptions of AI fall into what AI researchers would label as Artificial General Intelligence (AGI). AGI is still pure fiction; we have no ability to create it yet. The world thinks of AI as AGI-like capabilities, even for corporate leaders. It should be able to do all things.
Because literature shapes our perception, what we’ve seen is the concept of AI that has evolved from “something that could think for itself” into “something that is all knowing”. And this is perhaps at the heart of the mismatch when it comes to understanding AI: the false equivalence between intelligence and knowledge. And the reality is, AI is not a font of knowledge. Knowledge is contextual; it continues to evolve and reshape. And AI mimics intelligence; it does not truly reason from first principles. It’s not connecting the dots, but rather relying on dots that have already been connected before. And so when someone ask questions like: “Why can’t AI create data for my model; why can’t AI tell me what features will be most predictive; why can’t AI figure out the cybersecurity flaws in my system?” they believe AI has ALL the answers to use intelligently, but it simply does not.
Training isn’t Experience
In the aforementioned Dwarkesh Patel’s podcast, Richard Sutton had argued that humans learn more through experiences than mimicry. And this limits AI because AI is based on rote learning. However, Dwarkesh seems to believe that since large language models (LLM) have been trained on the world’s corpus of knowledge, it was, in essence, equivalent to having learnt through the collective human experience. This is another false equivalence: that sufficient rote learning might equate to experience-based learning. Memorising a bunch of facts and information does not imply comprehension. Comprehension comes when those facts and information are put into service and reviewed against their expected outcomes … that’s the power of experience, as Sutton opined.
This has implications to leaders looking to leapfrog their organisations through AI transformation, bypassing the need of building up core competencies in data analytics and data science. They believe they can acquire “expertise” with AI. The reality is that AI is brittle and less agile in a dynamic and rapidly changing world. AI is not an expert because expert-level competency comes from experience. There are no shortcuts for organisations to achieve data and AI competencies.
The Act of Generalisation
Another thing that seems to befuddle most people is the concept of generalisation. In his podcast with Sutton, Dwarkesh Patel was insistent that current LLMs (i.e. Gen AI) are able to generalise beyond the areas in which they’ve been trained for. He cites the ability for LLMs to solve Maths problems as evidence of that. But is it really solving Maths problems based on axiomatic first-principles approaches, or is it just lucky because it managed to break the problem down into subcomponents that have recognisable equivalents in the LLM’s training dataset? Researchers agree that it’s very much the latter.
Generalisation is a key unique human ability that we mostly take for granted. It is not dependent on stochastic, which is the underlying basis of all LLMs. And this inability for AI to generalise also means that it cannot innovate by recombining existing knowledge, it cannot transfer knowledge and best practices from one domain to another. For leaders looking to build distinct competitive advantage through the use of AI, this poses a challenge since AI is not a broad reusable asset. It’s not horizontally scalable, which is a key feature for a business’ long-term growth and sustainability.
Conclusion
AI literacy is so much more than just training employees to use Gen AI prompts as Citi and Accenture have announced. That’s the equivalent of teaching employees how to access the internet or use email; it isn’t transformative. To engage in meaningful discussions on where AI could fit into their job scopes, employees need a deeper understanding of what AI really is.
