UX Analytics Need Not Be So Complicated
Persona-based and user-shadowing needs to give way to more objective data-driven methods of sensemaking.
Background
My most popular article in Dec 2023 was one about the problem with UX analytics. I took issue with UX designers’ over-reliance on surveys to collect data, and the uselessness of feedback mechanisms like 5-star ratings. The article generated feedback and discussion on LinkedIn on both sides of the aisle; so I clearly struck a nerve. I recently had the opportunity to follow-up and deepen some of the thinking when an existing client, having read the article, reached out and asked me to come talk to them about their own approach to UX measurements. And so I’m dedicating my 21st weekly article to expanding on my thinking around UX analytics.
(I write a weekly article on bad thinking and bad practices in data analytics / data science which you can find here.)
Deconstructing UX
In my previous UX Analytics article, I shared that one my clients defines UX as ‘utility’, ‘usability’, ‘appeal’ and ‘trust’. This was further reinforced when I recently read an interesting article that highlighted why utility is the most important aspect of a product. It talked about the utility of McDonald’s milkshake as a “meal on the go” for long-drive commuters. It talked about the sheer market dominance of Excel (spreadsheet software) and Pro Tools (digital music production and editing software) despite their non-intuitive user interfaces and steep learning curves. When a solution has high utility (i.e. addresses a range of needs), users will forgive, or even not care about, other friction points. The idea of utility is linked to the concept of “jobs to be done”.
When we use the word “User Experience” we think of it as some ephemeral emotional connection that arises when using the product. It could be experienced at the moment of use or as part of an extended journey including pre- and post-use. Wikipedia describes UX as including “a person’s perceptions of utility, ease of use, and efficiency. … User experience is subjective but the attributes that make up the user experience are objective. … ISO 9241 definition of UX includes all the users’ emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviours and accomplishments that occur before, during, and after use.”
UX designers chase this concept by having focus groups and user-shadowing. They learn in school to use ethnographical methods to uncover ‘insights’ about the product use. I argue that this is outdated. In speaking with the said existing client recently, I proposed a simpler and more structured way to measure UX:
- Define utility as the attempted use of a product’s set of functionalities. We then measure what % of the functionality set is being utilised (attempted). The higher the %, the higher the utility value extracted for the user. (To properly define utility, a product manager needs to inventorise the set of product functionalities. At its core, a functionality is a capability of the product to accomplish a task or “job to be done”. This obviously requires thoughtful consideration.)
- Define usability as the success rate of using a product’s set of functionalities. We measure the % of attempts leading to successful usage of a given functionality. The higher the %, the better the usability.
Developing these measure allows us to shift the UX needle by providing answers to whether the existing product utility is high (correlated with high switching cost) and therefore giving the product owner a retentive advantage. The measures can provide answers as to whether the introduction of new functionality increases product utility, and therefore, whether it matters.
By measuring Usability separately for New and Existing users of the product, we can more objectively figure out if the product is not intuitive to use If there is a large gap in Usability between New and Existing users. Similarly, by measuring Utility separately for New and Existing users of the product, we can develop share-of-wallet benchmarks to drive awareness and ‘education’ initiatives for New users.
Conclusion
Given the ability to data-instrument most products today, particularly digital products, usage behaviour can be made more objective and reliable. We should not be relying on ethnographic methods to figure out HOW users are using a product (as UX designers have been taught to do). Instead, we should be using those ethnographic methods to uncover the mental states of WHY the needs exist, and WHAT motivates a user to select this product.
