The Use and Abuse of Polls as Predictive Tools

The 2020 presidential election is finally over. Among the enduring stories of the election

cycle was that the pollsters again got it wrong.  Specifically, in the closing week or so of the election Real Clear Politics documented polls from the Economist, Quinnipiac, NBC/Wall Street Journal, Survey USA, CNN, and Fox which predicted the national popular vote to have Joe Biden winning over Donald Trump by 10, 11, 10, 8, 12 and 8 points respectively.  In reality, Biden’s final national popular vote lead over Trump was 4.4%.  These errors are on top of claims that in 2016 pollsters and prediction machines such as FiveThirtyEight were wrong in not seeing that Trump would win.  Up to Election Day, FiveThirtyEight gave  Clinton a 72% chance of winning.

There may be a crisis in polling, but much of it has to do with a failure to understand survey research,  the  employment of bad polls, and the misuse and interpretation of them.

Remember first that polls or surveys are not meant to be predictive tools.  They are snapshots in time regarding what a statistical sample of people think about an issue.  This is one of the most fundamental errors that analysts, the media, and the predictive machines make.  When a survey is done it is predicated upon the answers individuals give  at the time they are surveyed.    They do not tell us what they are going to think in two weeks, they do not tell us what  people who are undecided are going to believe, and they do not tell us how many  individuals are actually included in the entire population of those who hold similar opinions or may actually vote.  All of these matters are predictive issues which polls cannot do.

Many in the media also simply do not understand statistics.  When a poll says that it  has a confidence level of .05 or 95% that does not mean it is 95% certain that this poll is an accurate prediction of what the final results will be on election day.  Many seem to think that.  The  confidence level refers to the fact that a pollster believes that a poll has a 95% chance of being an accurate random sample of the population being surveyed at that time.  Again, this is not a prediction for the future, but a statement about the current poll, and it also recognizes that there is a 5% or one-in-twenty chance the poll does not accurately represent the population it wants to survey.  This suggests that even a good pollster can get it.  Thus, some polls are not valid when done, and many are consistently not reliable over time and should simply be discounted or ignored.

Polls also vary in terms of what is called their margin of error.  The margin of error reflects  the size and composition of the sample done.  Surveys might report margins of error of plus or minus two, three, four, or more points.  The smaller the margin of error generally the better.  A poll saying Biden is leading Trump by seven percentage points, plus or minus a margin or error of three points, means the lead could be four or 10 points.  In tight elections, especially at the state level where presidents are selected due to the electoral college, leads of one or two point points with margins of error of three points are still technically correct even if the predicted winner loses.

Traditionally surveys used confidence intervals to assess these margins or error but increasingly some are using what are called Bayesian Credibility Levels.   They are not the same thing. 

Credibility Levels are used in  non-random samples  and  assess the probability that a sample reflects a  pollster’s predetermined  sample composition.  The use of non-random samples and Bayesian Credibility Levels opens up new sources of bias and inaccuracy in polls.  This might include mis-estimating some voters, such as non-college-educated males whose turnout was greater than these surveys assumed in the last two presidential election cycles. Phrased another way, confidence levels assess the probability that a sample is representative of  the real population. Credibility level assesses the probability the survey sample matches a  pollster’s predetermined belief of what it should look like. The American Association for Public Opinion Research has cautioned against this increasingly popular survey methodology, perhaps for good reasons.

Remember also that national polls for presidential elections are effectively meaningless.  We do not elect presidents with a national popular vote but with the electoral college that  makes it 50 separate state contests.  In the critical swing states of 2020 such as Georgia, Michigan, Pennsylvania, and Wisconsin, polls there predicted close races and once all the votes came in—not just those on election day and reported that night—polls in those states were accurate and the final results came within the accepted  margin of error.   Everyone seems to have forgotten this.

There are also other problems with polls that predict voting.  One needs to think about the questions asked, assumptions about who will show  up to vote, how many are undecided and when and if they will make up their mind.  Finally, yes in a world where no one picks up their  phone anymore it is hard to do samples, but if one is willing to spend the time and money and increase sample sizes, one can still get accurate polls.   The issue is willingness to commit the effort to doing it right.

Given the above and recognizing that polls are not predictive tools, we can see the fundamental flaws with tools such as FiveThirtyEight.  They take a collection of all polls—good and bad—average them, and then makes a statistical prediction of what is likely to happen in an election.  Phrased otherwise, they take instruments not meant for prediction and which already have statistical assumptions in them which might not be accurate, and then use them to make statistical predictions for a future event.  All this is highly questionable.  Then analysts still ignore the fact that a prediction machine that says it is 70% likely something will happen means, even by its own estimate, a 30% chance of being wrong.  Error compounds error, statistical assumptions multiple upon themselves, and a failure to understand statistics yields the belief that the polls simply have it wrong.

Polling is an exercise based on probability and chance.  It is not a perfect predictive tool that can foresee the future in a clairvoyant way. If  viewed in this light we find the crisis in polling and predictive machines is less about traditional polling and more about their misuse and abuse.

Why Nate Silver is Wrong: The Limits of Political Money Ball and Why Trump may Win

It’s time to admit it–Trump may win.
Increasing and begrudgingly the establishment politicians, pundits, and analysts are beginning to realize that Trump may actually win the presidency. Nate Silver, whom too many people put too much political stock in, is now saying that Clinton is favored 60-40% to win, down from dramatically larger percentages even just a couple of weeks ago.  It’s nice to see that Silver finally is getting closer to my assessment which has said Clinton has a 55-60% chance of winning.  But even then, I may be exaggerating her chances and would put it at 50%+–barely break even.
Nate Silver came to fame with applying the logic of money ball to politics–successfully using his algorithms to call 99 of 100 states in the last two presidential elections.  Silver is smart, but it would not have taken an Einstein to call at least 90 of the 100 states.  This is what the logic of my book Presidential Swing States is all about–showing how because of the Electoral College, partisan voting, and party alignments, the elections were over before they started in all but ten states.  Moreover, in the last two election cycles, one could have also eliminated a couple of other states from the swing state category, giving one about 94 states that would have been easy to predict.   Throw in the relative stability of polling and getting to 99 is not so hard.
Why is all this worth mentioning in connection with Trump and the 2016 elections?  First,  despite 2016 being a unique election year there are still many forces that make it a relatively normal election that again is reducing the election to only a handful of swing states.  Still in play are the core ten that have consistently been in play for the last seven election cycles, such as Ohio, Florida, North Carolina, and Iowa, but additionally a few others such as Michigan and Pennsylvania may now be  flip-able.  There is evidence that partisanship is still a factor driving how people vote with few real swing voters moving from party to party.  Voters are coalescing around Clinton to some degree but more so for Trump, thereby producing a normal election pattern that suggests is it still a few swing voters in a few swing counties in a few swing states that will determine who gets to 270 electoral votes and wins the presidency.  All this bodes well for the Nate Silvers of the world.
Additionally when one looks at Trump versus Clinton traditional wisdom hands it to Clinton.  Until recently ahead in the polls, she has a better run campaign, more money, and has insulted far few people than Trump.    Yet this is where the uniqueness of 2016 kicks in, and where the limits of political money ball appear.
First, there is no such thing as an electoral college lock for the Democrats who think demographics is destiny.  Statistically voters may be presupposed to vote a certain way but you need to get them to vote.  Trump supporters are passionate and will show up, Clinton’s are not.  She relies on many voters who are mercurial at best when it comes to vote and she has done little to address the lack of enthusiasm many have for her.  She has yet to seal the deal with the Sanders people and liberals, simply assuming that running to the center as a Republican much like here husband did will result in these people having no where else to go and therefore they will vote for her.  2016 and the Millennials are very different from 1992 and the Baby Boomers.  Even African-Americans who loved  Obama in 2008 and 2012 may not come out the same way in 2016.
Second, polling is more complicated now than before. Cell phone technology, polling costs,  defining likely voter, and other issues all complicate this years predictions.  Many media outlets are cutting costs on polls.  Take for example the September 18, 2016 Star Tribune poll  with results from 625 respondents and landlines constituting 69%.  A good poll should have at least 1,000 respondents and nearly 70% cell phone.  This is a flawed  poll.  Silver’s predictions are only as good as the polls and he has blown several predictions this year, consistently over-estimating Clinton’s strengths.  Face it, Clinton often polls better than she performs, Trump performs better than polls.
Third, what political money ball misses are three important factors–candidate quality, mood of a country, and the politainment quality of American politics.  No matter how good a campaign some candidates are simply not good.  Clinton is a weak candidate and does not resonate well.  Factors such as likeablity are missed in political money ball.  Yes, Trump too is a weak candidate, but he has the benefit of it being an anti-establishment year at a time when Clinton is the poster child for the establishment.  And Trump understands the politainment aspect of contemporary politics that is increasing post-truth (candidates do not tell the truth and the public does not expect it), post-rational, post-issues, and simply pop culture sound-bite driven.  Image is everything, content is nothing.
Put this altogether and the traditional political pundits, politicians, analysts, and Nate Silvers of the world are missing a tremendous amount about politics in 2016.    They and traditional political scientists also miss the importance of how politics is moving marginal numbers of swing voters in a few swing counties in a few swing states and therefore the issue is not how Trump or Clinton appeal to large numbers of people but only to move a few people.  Aggregate analysis misses subtle shifts.
Right now logic dictates Clinton still should win but the reality may be that the public should prepare for a Trump presidency because there are many reasons to think they he will win.

Postscript:  The Star Tribune poll is flawed along the lines of its previous polls that predicted a Dayton blowout over Emmer in the 2010 gubernatorial race.  Notwithstanding the poll's problems there is no doubt the race for president in Minnesota is closer than some think.  Trump plays well in an Iron Range ready to flip to the GOP and I can see it electing Mills or Nolan.  The same  could affect the first district race Waltz v Hagedorn and even Craig v. Lewis and the control for the MN House and Senate.  Clinton's weaknesses have potential down ballot issues and the DFL's effort to go to court last week to get Trump off the ballot may have given new resolve to Republicans.  Add to that the ISIL connection to the St Cloud attack and how the public sees in some polls security and law and order as a Trump issue and one can see Minnesota as potentially competitive at the presidential level.  While possibly not yet a swing state, I see many reasons based on the conclusions of my recent book why Minnesota is becoming a swing state, perhaps even as early as 2016.