The Use and Abuse of Polls as Predictive Tools

The 2020 presidential election is finally over. Among the enduring stories of the election

cycle was that the pollsters again got it wrong.  Specifically, in the closing week or so of the election Real Clear Politics documented polls from the Economist, Quinnipiac, NBC/Wall Street Journal, Survey USA, CNN, and Fox which predicted the national popular vote to have Joe Biden winning over Donald Trump by 10, 11, 10, 8, 12 and 8 points respectively.  In reality, Biden’s final national popular vote lead over Trump was 4.4%.  These errors are on top of claims that in 2016 pollsters and prediction machines such as FiveThirtyEight were wrong in not seeing that Trump would win.  Up to Election Day, FiveThirtyEight gave  Clinton a 72% chance of winning.

There may be a crisis in polling, but much of it has to do with a failure to understand survey research,  the  employment of bad polls, and the misuse and interpretation of them.

Remember first that polls or surveys are not meant to be predictive tools.  They are snapshots in time regarding what a statistical sample of people think about an issue.  This is one of the most fundamental errors that analysts, the media, and the predictive machines make.  When a survey is done it is predicated upon the answers individuals give  at the time they are surveyed.    They do not tell us what they are going to think in two weeks, they do not tell us what  people who are undecided are going to believe, and they do not tell us how many  individuals are actually included in the entire population of those who hold similar opinions or may actually vote.  All of these matters are predictive issues which polls cannot do.

Many in the media also simply do not understand statistics.  When a poll says that it  has a confidence level of .05 or 95% that does not mean it is 95% certain that this poll is an accurate prediction of what the final results will be on election day.  Many seem to think that.  The  confidence level refers to the fact that a pollster believes that a poll has a 95% chance of being an accurate random sample of the population being surveyed at that time.  Again, this is not a prediction for the future, but a statement about the current poll, and it also recognizes that there is a 5% or one-in-twenty chance the poll does not accurately represent the population it wants to survey.  This suggests that even a good pollster can get it.  Thus, some polls are not valid when done, and many are consistently not reliable over time and should simply be discounted or ignored.

Polls also vary in terms of what is called their margin of error.  The margin of error reflects  the size and composition of the sample done.  Surveys might report margins of error of plus or minus two, three, four, or more points.  The smaller the margin of error generally the better.  A poll saying Biden is leading Trump by seven percentage points, plus or minus a margin or error of three points, means the lead could be four or 10 points.  In tight elections, especially at the state level where presidents are selected due to the electoral college, leads of one or two point points with margins of error of three points are still technically correct even if the predicted winner loses.

Traditionally surveys used confidence intervals to assess these margins or error but increasingly some are using what are called Bayesian Credibility Levels.   They are not the same thing. 

Credibility Levels are used in  non-random samples  and  assess the probability that a sample reflects a  pollster’s predetermined  sample composition.  The use of non-random samples and Bayesian Credibility Levels opens up new sources of bias and inaccuracy in polls.  This might include mis-estimating some voters, such as non-college-educated males whose turnout was greater than these surveys assumed in the last two presidential election cycles. Phrased another way, confidence levels assess the probability that a sample is representative of  the real population. Credibility level assesses the probability the survey sample matches a  pollster’s predetermined belief of what it should look like. The American Association for Public Opinion Research has cautioned against this increasingly popular survey methodology, perhaps for good reasons.

Remember also that national polls for presidential elections are effectively meaningless.  We do not elect presidents with a national popular vote but with the electoral college that  makes it 50 separate state contests.  In the critical swing states of 2020 such as Georgia, Michigan, Pennsylvania, and Wisconsin, polls there predicted close races and once all the votes came in—not just those on election day and reported that night—polls in those states were accurate and the final results came within the accepted  margin of error.   Everyone seems to have forgotten this.

There are also other problems with polls that predict voting.  One needs to think about the questions asked, assumptions about who will show  up to vote, how many are undecided and when and if they will make up their mind.  Finally, yes in a world where no one picks up their  phone anymore it is hard to do samples, but if one is willing to spend the time and money and increase sample sizes, one can still get accurate polls.   The issue is willingness to commit the effort to doing it right.

Given the above and recognizing that polls are not predictive tools, we can see the fundamental flaws with tools such as FiveThirtyEight.  They take a collection of all polls—good and bad—average them, and then makes a statistical prediction of what is likely to happen in an election.  Phrased otherwise, they take instruments not meant for prediction and which already have statistical assumptions in them which might not be accurate, and then use them to make statistical predictions for a future event.  All this is highly questionable.  Then analysts still ignore the fact that a prediction machine that says it is 70% likely something will happen means, even by its own estimate, a 30% chance of being wrong.  Error compounds error, statistical assumptions multiple upon themselves, and a failure to understand statistics yields the belief that the polls simply have it wrong.

Polling is an exercise based on probability and chance.  It is not a perfect predictive tool that can foresee the future in a clairvoyant way. If  viewed in this light we find the crisis in polling and predictive machines is less about traditional polling and more about their misuse and abuse.