Polling Fails!
The problem with surveys.
Background
I knew Trump would win. I said as much during my recent monthly catch-up call with ex-colleagues (former regional and country heads of Analytics in my previous place of employment). I didn’t trust the polls. I based my prediction on some simple observations about the US context:
- More people vote when times are uncertain, and times are extremely uncertain for the less-educated and lower/middle class. So more Republican voters than Democratic voters.
- Trump stood for something consistently, Harris stood for ???. (I am not a fan of Trump, but calling it as I see it.)
Of course, with binary outcomes, it’s easy to post-rationalise our “brilliance” in seeing the patterns. I was probably just lucky in my assessment. But this article isn’t about that. Rather, this article is about why we shouldn’t be putting our trust in polls and surveys, regardless of how technically brilliant they may appear. And so I dedicate my 64th article to unpacking the problem with survey data.
(I write a weekly series of articles where I call out bad thinking and bad practices in data analytics / data science which you can find here.)
False Comfort in Polls & Surveys
The 2024 US presidential election once again revealed the inadequacies of polling data. (Polls are just short-form surveys.) This is the 3rd consecutive US presidential election poll that has been significantly out of whack! Out of the gate, the polling data showed Harris having a strong lead against Trump, and almost every single poll showed her leading just before Nov 5, albeit with margins closing. But as we all know, Trump completely crushed Harris. So obviously, the polling data was not reflective of reality.
Specifically with US election polling, polling response rates have fallen to 1% in some states. There is simply no way to claim statistical representation at those dismal response levels. So what the pollsters do is to “reshape” the limited response data by making certain segments larger or smaller to fit their underlying beliefs of what fair representation should look like. But these assumptions are often fraught, even though they may be well-intentioned. The lesson here is that you can’t post-fix representation; it needs to be addressed as source.
Responses to surveys, in general, have fallen sharply over the last couple of decades. For example, the response rates to the famous Pew Research Center telephone surveys have declined from 36% in 1997 to 9% in 2016 to 6% in 2018, and was discontinued soon after. There is also the increased phenomenon of non-response bias — i.e. non-random systematic reasons for those unable or unwilling to respond to your survey; they may be giving off a distinctive set of information signals. All of this leads to increasing mis-representation.
While I don’t have the answers, it would seem to me that we will increasingly need to move away from response data and towards observation data as the source for “polling”.
Valid Representation
Now, this problem of mis-representation exists in many corporate surveys as well — e.g. employee or customer satisfaction surveys. Two issues abound here: (a) low or obviously skewed responses, and (b) the lack of validation. We talked about point (a) above leading to mis-representation. But point (b) is often overlooked. In the case of the US presidential elections polling, the accuracy of the polls is finally validated with the election results, allowing them to hypothesise where they might have gone wrong in their polling set-up. But because it’s a long 4-year cycle, there is both insufficient frequency to get the corrections right, and the likelihood of ground shifts between cycles. Nonetheless, there is some validation at play. However, for corporate surveys like employee or customer satisfaction, there is no real validation. What does a 70% satisfaction score mean versus an 80% satisfaction score? How do we know if it has truly moved up? What would have to be true or manifested if the satisfaction score truly moved up? If we haven’t yet mapped these out, then running a satisfaction survey is just a vanity project with no real value being created.
Polling and survey data are ultimately estimates. We must therefore establish ways to either validate or triangulate the estimates.
No Wisdom In Crowds
The last few decades have seen aggregation and simulation being utilised in the US presidential election polling. It is anchored on the belief in the often touted “wisdom of the crowd”. For example, Nate Silver’s FiveThirtyEight uses polling aggregation and simulation. But still got the election results wrong.
Here is the common-sense explanation for why aggregation doesn’t work. In fact, the argument has equivalence in predictive modelling using machine learning. The “wisdom of the crowd” is based on the cancelation effect in error rate by combining multiple independent estimates. But it works only if all the estimates are based on the same information signals; the variance in errors being derived from their different interpretation of the information signals. But in the case of the US presidential election polling, each polling method uses a different set of data points and information signals. There is therefore no cancelling effect when aggregated, and in fact, the error rate is likely unknown. There is no crowd wisdom here.
However, there may be wisdom in thinking out of the box. I recently read about the “neighbour polls” in which you ask a prospective responder: “Who do you think your neighbour will vote for?”. It seems this method more accurately predicted the outcome of the 2024 US presidential election. While not validated, this nonetheless reveals that there is still room for innovation in polls and surveys.
Conclusion
Those who have been following my articles would have noted that I’m not a fan of survey data, and would be hesitant to make decisions based on them. Non-representation and validity are my chief concerns, caused by falling response rates, non-response bias, and lack of close-loop feedback.
Rather, I’m a fan of observational data, and even outcome data — e.g. asking the question “What outcome would have to be true if my survey data had an XXX value?”
There will of course be times when we would have to run surveys, but it should be seen as supplementary data rather than primary data.
