Public opinion polls are perplexing. The same is true nationally as in Minnesota and the polling on the state’s two constitutional amendments demonstrate some of the difficulties in doing survey research and public opinion polls these days.
On Sunday the Star Tribune released survey results on the two constitutional amendments. It found that with the Marriage Amendment (to further ban same-sex marriage even more than it is banned in the state by Court decision and law) support for it was at 49%, opposed at 47%, and 4% undecided. For the Elections Amendment to require photo ID at the ballot (and with that effectively to sharply limited if not eliminate election day registration), support for it was a 52%, opposition 44%, and undecided at 4%.
What are we to make of these polls? Are the numbers accurate? Perhaps, but the Star Tribune poll is in need of some qualifications and corrections.
First, look at the poll. It has a margin of error of 3.5%. In itself this margin of error is not bad, but the poll also indicates that for subgroups such as party affiliation the margins of error are larger than that. Again, no surprise that a poll of only 800 respondents would have more error in tabulating results for subgroups. But what is more interesting is the partisan breakdown of the poll-- 41 percent Democrat, 28 percent Republican, and 31 percent independent or other–and the landline versus cellphone population–80% versus 20%. But of these numbers could skew the survey in critical ways.
Consider first partisan affiliation. The survey may have overpolled Democrats and undepolled Republicans. After the 2008 elections (a great year for Democrats), my estimate was that the state was about 39% DFL, 33% GOP, 10% IP (Independent Party), and 18% other. Following the 2010 election (a great year for Republicans), my estimate was 38% DFL, 34% GOP, 12% IP, and 16% other. It seems hardly likely that DFL affiliation in the state today has risen to a level exceeding the 2008 estimates, even considering disapproval with the Republican legislature and Congress. Similarly, after 2010, it is unlikely that GOP support has dropped to 28%. More likely numbers are something along the lines of 38% DFL, 34% GOP, 11% IP, and 15% other.
Why is the partisan adjustment important? The poll suggests significant partisan polarization for both amendments, with 73% of DFLers opposing the marriage amendment and 71% of GOPers supporting. Similar partisan cleavages also exist with the Elections Amendment. If this is true, take the marriage Amendment support at 49% and opposition at 47%. If DFLers are overpolled by 3% and GOP underpolled by 6%, and if about 3/4 of each party votes in a partisan way, I would subtract about 2.25% from opposition (3% x .75) and add 4.5% to support (6% x .75) and the new numbers are 53.5% in support and 44.75% against. This is beyond margin or error.
If one applies the correction to the Elections Amendment there is about an 80% DFL opposition to it and a similar 80% GOP support for it. Then the polls suggest approximately 56.8% support it and 41.6% oppose.
Now there is a second adjustment to the poll–landline versus cell phone users. The poll had 80% landline. This overpolls this type of user. Some estimates are that as many as 40% of the population, especially those who are younger, rely exclusively on cell phones. Older voters are more likely to support the Marriage and Elections Amendment. Again, there seems to be an age divide with about 60%of older voters supporting the Marriage Amendment and about 60% of younger opposed to it. This skewing needs to be corrected but there are two complications here. First, younger people are less likely to vote than older people. Thus, even if the survey overpolled elderly, the results also have to consider undervoting by younger voters. Any serious correction to the skewing by landline and then by failing to correct for younger people not voting leads to concerns about the survey accuracy, even if it claims to have surveyed likely voter. At the least landline or older voters were oversurveyed by perhaps 10% and younger voters not voting may affect the survey by several percent. Nationally, the gap between elderly and younger voters is as high as 10%, in Minnesota it is much less but still significant. I have no good tool at this time to correct for these two variables, but think they might about cancel one another out. My intuition though is to contend that younger cellphone users who will actually vote are underpolled by at least an overall 2%. If that is true, call it 51.5% of the Marriage Amendment and 46.75% against. For the Elections Amendment, now call it 54.8% in support, 43.6% against.
Two other corrections now need to be made. Support for anti-gay initiatives is underpolled compare d to final election returns. Why? People lies to pollsters. In the 2008 California Prop 8 battle, the last survey revealed 47% supporting repeal, with the final election results being 52.24% This is a difference of 5.24%. In Maine in 2009 the last poll prior to repealing its gay marriage legislation yielded an underpolling of 4.9% comparted to the final election results. Yes public opinion might shift, but my estimate is that polls underestimate support for anti-gay rights legislation by about 5%.
Thus, 55.5% in favor and about 42.% opposed to the Amendment. I am assuming that most of the 4% undecided in the poll vote yes.
However, there is also an issue about constitutional amendment voting in Minnesota. If an individual votes in an election but fails to vote on the amendment then the failure to vote is counted as a no vote. What percentage of the electorate undervotes on constitutional amendments? Going back to 1988 and looking at the last 12 amendments, the undervote averages 4.84%. It ranged from a high of 7.68% to a low of 0%. Why is this important? One needs potentially to subtract 5% from any support for constitutional amendments because some people will vote in the election but not on the amendments. It is possible that there will be a similar 4.84% average undervote on one or both of the amendments, but it is also possible it will be lower because both are politically salient and controversial. My guess is an undervote of about 2%. This means subtract 2% from support and then add 2% to opposition.
So where do we stand after all my corrections?
Marriage Amendment
Yes at 53.5%, with No at 44%.
Elections Amendment
Yes at 52.8%, with No at 45.6%.
My survey corrections do not include the impact of early voting in Minnesota that has already commenced. This may complicate messaging and moving people who have already voted. However, my argument is that the current polls over/underestimate real partisan affiliation, underpoll younger people (cell phone users), fail to calculate undervoting by the young, anti-gay sentiment, and neglect constitutional amendment undervoting. My result thus suggest a closer vote on the Elections Amendment than polls suggest and a larger margin on the Marriage Amendment. Of course, opinion may change and that is why there are elections. Let us see what happens on November 6.
Showing posts with label surveys. Show all posts
Showing posts with label surveys. Show all posts
Sunday, September 23, 2012
Friday, July 27, 2012
And the Winner Is....Predicting the 2012 Presidential Election
Note: Several months ago I published an article predicting the 2012 presidential race. On July 26, 2012 I was the keynote speaker at the Coalition of Greater Minnesota Cities Summer Conference/Annual Awards Dinner where I updated my prediction. Here is a summary of my talk and prediction.
Presidential public opinion polls are perplexing. Some polls put Obama ahead of Romney, some say the race is tightening. Others find Obama ahead in critical swing states while others describe swing voters as perhaps moving toward Romney.
Ignore all of these polls. The only three numbers I think that are important are these: 10/10/270. Let me explain.
You should ignore the polls first because they are snapshots in time, more than almost four months before the November elections. Too many things can happen–a collapsing economy, war in Syria, gas prices, campaign gaffes–which can impact the race in the next few months. Ignore the polls also because they are national opinion polls reflecting aggregate opinion across the county. As Florida in the 2000 presidential election taught us, one can win the popular vote in a presidential election (as Al Gore did) but still lose the presidency in the Electoral College. What matters most is winning 270 electoral votes. The presidency is a battle not across 50 states but in 50 states. In contrast to the Republican presidential contest that has turned less from winning individual states than to amassing delegates, the general election in November is really one of winning enough electoral votes to reach 270–a majority of the 538 electoral votes at stake.
What complicates the race to 270 is that with the exception of Maine and Nebraska, the remaining 48 states plus the District of Columbia award their electoral votes on a winner-take-all basis. Whichever candidate wins a plurality of the popular vote in a particular state wins all of its electoral votes. Thus the general election is both about winning states and amassing delegates.
Why is all of this important? Simply the presidential race is over in 40 states. There are some states that are reliably Democratic or Republican. No one seriously thinks a Republican is going to win New York and even though Mitt Romney is its former governor, neither he nor any other Republican has a prayer to win Massachusetts. Conversely, even though Romney’s recent bad news was that he could not prevail in Alabama and Mississippi, the good news for him and Republicans is that no Democrat is going to win there. The race for the presidency is simply over in these states and Democrats in Texas and Republicans in California might as well do something else besides casting presidential votes in November.
Barack Obama is reasonably assured of winning California (55 electoral votes), Connecticut (7), Delaware (3), Hawaii (4), Illinois (20), Maine (4), Maryland (10), Massachusetts (11), New Jersey (14), New York (29), Oregon (7), Rhode Island (4), Vermont (3), Washington state (12) and Washington, D.C. (3). And despite protests from Republicans that Minnesota (10) is competitive, that is a fairy tale. If Minnesota is a swing state then it is truly over for Obama. Don’t look for the candidates or TV ads to be here come October. Thus, Obama starts with 15 states (plus D.C.) and 196 electoral votes.
Conversely, Mitt Romney or any other Republican nominee is reasonably assured of winning Alabama (9), Alaska (3), Arkansas (6), Georgia (16), Idaho (4), Kansas (6), Kentucky (8), Louisiana (8), Mississippi (6), Montana (3), Nebraska (5), North Dakota (3), Oklahoma (7), South Carolina (9), South Dakota (3), Tennessee (11), Texas (38), Utah (6), West Virginia (5) and Wyoming (3). Moreover, it was luck in 2008 that Obama won Indiana (11) and that is not in the cards this year. This is a total of 21 states and 170 electoral votes.
Initially, this means a total of 14 states, with 172 electoral votes, are potentially in play. These swing states will determine the outcome of the election and within them, swing voters–roughly 10-15% of the voters–will make the difference. Thus, the battle for the presidency is really over what a handful of swing voters do in 14 swing states. These states are: Arizona (11), Colorado (9), Florida (29), Iowa (6), Michigan (16), Missouri (10), Nevada (6), New Hampshire (4), New Mexico (5), North Carolina (15) Ohio (18), Pennsylvania (20) Virginia (13), and Wisconsin (10).
Now some of the states are debatable as swing. It is a long shot for Democrats to win Arizona even with a strong Hispanic turnout. In hopes of winning North Carolina Democrats are holding their convention in Charlotte. But there is no evidence that there is a convention bump; look to the Republican National Convention in Minnesota in 2008 as rendering North Carolina a long shot. Similarly, Republicans consistently see Pennsylvania as one that they can win, but the Keystone State, as well as Michigan, remain more Democratic than Republican. The last time the GOP won Pennsylvania was in 1984 with Ronald Reagan. In the case of Michigan, Republicans running against the auto bailout seem to be a losing strategy. Thus, the original 14 state list could be reduced to ten, leaving Obama with 232 electoral votes, and a Republican nominee with 196. This leaves 110 electoral votes in contest.
These states are: Colorado (9), Florida (29), Iowa (6), Missouri (10), Nevada (6), New Hampshire (4), New Mexico (5), Ohio (18), Virginia (13), and Wisconsin (10).
If pressed I could even make the argument that it is over in Wisconsin (Obama) and Missouri (Romney) but will not.
In these swing states the percentage of undecideds ranges from 10% in Colorado, Ohio, Florida, Tennessee, Iowa, and Missouri, to about 7 percent in Nevada, with the overall undecideds in the country at 10%. Thus this is how I arrive at 10/10/270–Ten percent of the voters in ten states determine who gets to 270.
Now the question is who the undecideds are and what will move them. Politics is about moving marginals (swing voters) (politics as a bar fight) Consider who these voters are and the issues that concern them.
First they are the unemployed, struggling middle class affected by the economy gas prices, and unemployment. These are the white working class. Bad news for Obama is that he does not connect with them. Good news for Obama, they also do not connect with Romney.
The second group of swing voters include moderate women concerned by recent debates over reproductive rights, birth control, and family issues. These are the soccer moms. The swing to the right has alienated many of these women from the GOP and Obama and the Democrats seem to be enjoying an unusually large gender gap this year.
The third group of swing voters are young people under thirty. It is odd to call them swing voters especially since four years ago they came out strong for Obama. This time around they are nowhere near as excited by him as they were in 2008, mostly because of economic issues and the failure of Obama really to connect with them.. These voters should be part of Obama’s base but because of their unpredictable turnout it is apt to call them swing voters. If they do show up they will vote for Obama.
Potentially these three groups and issues overlap; making it difficult to decide which is the most important or will tip the balance in the election. But assume for minute that this presidential election is similar to many others in that it is a economic referendum on the incumbent–then it is the economy that is the main issue. How is the economy doing in these swing states? It is a mixed bag, with unemployment levels stagnating along with economic growth. Moreover, neither candidate seems to be doing a good job coming up with an election narrative except for saying “I’m not Obama” or “I’m not Romney.”
Having said all that, here is how I think the remaining states are tipping now.
Add to Obama’s 232 the following states: Colorado (9), Nevada (6), New Hampshire (4), New Mexico (5), and Wisconsin (10). This is 34 more electoral votes that gets him to 268, two short of the necessary 270.
Add to Romney’s 196 : Missouri (10), Virginia (13). This is 23 more electoral votes and gets him to 219,still 51 short.
Three states, Florida (29), Iowa (6), Ohio (18), total 53 electoral votes, and they are too close to call at this time. Romney needs all three states to win, Obama one state to win.
Unless this election is a replay of 1980 where disgust for the status quo and Carter was so strong that it tipped millions of swing voters in the last 72 hours to vote for Reagan, I do not see Romney presenting winning all three of these states. Obama wins at least one of these states, thereby ensuring his re-election. Obama wins with 274-321 electoral votes.
Presidential public opinion polls are perplexing. Some polls put Obama ahead of Romney, some say the race is tightening. Others find Obama ahead in critical swing states while others describe swing voters as perhaps moving toward Romney.
Ignore all of these polls. The only three numbers I think that are important are these: 10/10/270. Let me explain.
You should ignore the polls first because they are snapshots in time, more than almost four months before the November elections. Too many things can happen–a collapsing economy, war in Syria, gas prices, campaign gaffes–which can impact the race in the next few months. Ignore the polls also because they are national opinion polls reflecting aggregate opinion across the county. As Florida in the 2000 presidential election taught us, one can win the popular vote in a presidential election (as Al Gore did) but still lose the presidency in the Electoral College. What matters most is winning 270 electoral votes. The presidency is a battle not across 50 states but in 50 states. In contrast to the Republican presidential contest that has turned less from winning individual states than to amassing delegates, the general election in November is really one of winning enough electoral votes to reach 270–a majority of the 538 electoral votes at stake.
What complicates the race to 270 is that with the exception of Maine and Nebraska, the remaining 48 states plus the District of Columbia award their electoral votes on a winner-take-all basis. Whichever candidate wins a plurality of the popular vote in a particular state wins all of its electoral votes. Thus the general election is both about winning states and amassing delegates.
Why is all of this important? Simply the presidential race is over in 40 states. There are some states that are reliably Democratic or Republican. No one seriously thinks a Republican is going to win New York and even though Mitt Romney is its former governor, neither he nor any other Republican has a prayer to win Massachusetts. Conversely, even though Romney’s recent bad news was that he could not prevail in Alabama and Mississippi, the good news for him and Republicans is that no Democrat is going to win there. The race for the presidency is simply over in these states and Democrats in Texas and Republicans in California might as well do something else besides casting presidential votes in November.
Barack Obama is reasonably assured of winning California (55 electoral votes), Connecticut (7), Delaware (3), Hawaii (4), Illinois (20), Maine (4), Maryland (10), Massachusetts (11), New Jersey (14), New York (29), Oregon (7), Rhode Island (4), Vermont (3), Washington state (12) and Washington, D.C. (3). And despite protests from Republicans that Minnesota (10) is competitive, that is a fairy tale. If Minnesota is a swing state then it is truly over for Obama. Don’t look for the candidates or TV ads to be here come October. Thus, Obama starts with 15 states (plus D.C.) and 196 electoral votes.
Conversely, Mitt Romney or any other Republican nominee is reasonably assured of winning Alabama (9), Alaska (3), Arkansas (6), Georgia (16), Idaho (4), Kansas (6), Kentucky (8), Louisiana (8), Mississippi (6), Montana (3), Nebraska (5), North Dakota (3), Oklahoma (7), South Carolina (9), South Dakota (3), Tennessee (11), Texas (38), Utah (6), West Virginia (5) and Wyoming (3). Moreover, it was luck in 2008 that Obama won Indiana (11) and that is not in the cards this year. This is a total of 21 states and 170 electoral votes.
Initially, this means a total of 14 states, with 172 electoral votes, are potentially in play. These swing states will determine the outcome of the election and within them, swing voters–roughly 10-15% of the voters–will make the difference. Thus, the battle for the presidency is really over what a handful of swing voters do in 14 swing states. These states are: Arizona (11), Colorado (9), Florida (29), Iowa (6), Michigan (16), Missouri (10), Nevada (6), New Hampshire (4), New Mexico (5), North Carolina (15) Ohio (18), Pennsylvania (20) Virginia (13), and Wisconsin (10).
Now some of the states are debatable as swing. It is a long shot for Democrats to win Arizona even with a strong Hispanic turnout. In hopes of winning North Carolina Democrats are holding their convention in Charlotte. But there is no evidence that there is a convention bump; look to the Republican National Convention in Minnesota in 2008 as rendering North Carolina a long shot. Similarly, Republicans consistently see Pennsylvania as one that they can win, but the Keystone State, as well as Michigan, remain more Democratic than Republican. The last time the GOP won Pennsylvania was in 1984 with Ronald Reagan. In the case of Michigan, Republicans running against the auto bailout seem to be a losing strategy. Thus, the original 14 state list could be reduced to ten, leaving Obama with 232 electoral votes, and a Republican nominee with 196. This leaves 110 electoral votes in contest.
These states are: Colorado (9), Florida (29), Iowa (6), Missouri (10), Nevada (6), New Hampshire (4), New Mexico (5), Ohio (18), Virginia (13), and Wisconsin (10).
If pressed I could even make the argument that it is over in Wisconsin (Obama) and Missouri (Romney) but will not.
In these swing states the percentage of undecideds ranges from 10% in Colorado, Ohio, Florida, Tennessee, Iowa, and Missouri, to about 7 percent in Nevada, with the overall undecideds in the country at 10%. Thus this is how I arrive at 10/10/270–Ten percent of the voters in ten states determine who gets to 270.
Now the question is who the undecideds are and what will move them. Politics is about moving marginals (swing voters) (politics as a bar fight) Consider who these voters are and the issues that concern them.
First they are the unemployed, struggling middle class affected by the economy gas prices, and unemployment. These are the white working class. Bad news for Obama is that he does not connect with them. Good news for Obama, they also do not connect with Romney.
The second group of swing voters include moderate women concerned by recent debates over reproductive rights, birth control, and family issues. These are the soccer moms. The swing to the right has alienated many of these women from the GOP and Obama and the Democrats seem to be enjoying an unusually large gender gap this year.
The third group of swing voters are young people under thirty. It is odd to call them swing voters especially since four years ago they came out strong for Obama. This time around they are nowhere near as excited by him as they were in 2008, mostly because of economic issues and the failure of Obama really to connect with them.. These voters should be part of Obama’s base but because of their unpredictable turnout it is apt to call them swing voters. If they do show up they will vote for Obama.
Potentially these three groups and issues overlap; making it difficult to decide which is the most important or will tip the balance in the election. But assume for minute that this presidential election is similar to many others in that it is a economic referendum on the incumbent–then it is the economy that is the main issue. How is the economy doing in these swing states? It is a mixed bag, with unemployment levels stagnating along with economic growth. Moreover, neither candidate seems to be doing a good job coming up with an election narrative except for saying “I’m not Obama” or “I’m not Romney.”
Having said all that, here is how I think the remaining states are tipping now.
Add to Obama’s 232 the following states: Colorado (9), Nevada (6), New Hampshire (4), New Mexico (5), and Wisconsin (10). This is 34 more electoral votes that gets him to 268, two short of the necessary 270.
Add to Romney’s 196 : Missouri (10), Virginia (13). This is 23 more electoral votes and gets him to 219,still 51 short.
Three states, Florida (29), Iowa (6), Ohio (18), total 53 electoral votes, and they are too close to call at this time. Romney needs all three states to win, Obama one state to win.
Unless this election is a replay of 1980 where disgust for the status quo and Carter was so strong that it tipped millions of swing voters in the last 72 hours to vote for Reagan, I do not see Romney presenting winning all three of these states. Obama wins at least one of these states, thereby ensuring his re-election. Obama wins with 274-321 electoral votes.
Thursday, May 24, 2012
How to Read Polling Data
Confused by all the polls and surveys taking place? Who is ahead or behind? I was interviewed by Profnet for this article on doing polls and it is worth reprinting here.
Dear Gracie,
With elections approaching, I've seen a lot of polls in the news recently. How do we know if the polls are accurate or biased?
Puzzled by Polls
Dear Puzzled by Polls,
Three ProfNet experts provide some insight: What You Need to Understand About Polls
"Creating and fielding a poll is not something that just anyone can do at the drop of a hat," says Jason Reineke, associate director of the Middle Tennessee State University (MTSU) Poll, which is a statewide, biannual poll of Tennesseans; as well as the university's assistant professor of journalism.
"It is both an art and a science, and the people who do it well usually have extensive training and expertise," continues Reineke. "Like a journalist, lawyer or medical doctor, being a pollster is a profession."
Polls are snapshots in time and not predictive tools, explains David Schultz, law and graduate school professor at Hamline University's School of Law, and editor of the Journal of Public Affairs Education. For example, polls conducted today about the presidential elections are not necessarily indicative of what will happen in November.
"A common problem with political polls is that they are often fielded by one party to support its agenda," adds Bob Clark, president of 24K Marketing.
Some polls are better than others, but the value of a poll can be better determined by the goals that it was designed to address, rather than one-size-fits-all rules, says Reineke. "Nonetheless, there are some standards that can be applied across most polls."
Transparency
Pollsters should freely and honestly report information about the poll's funding, affiliation, methodology, data and analysis, explains Reineke.
"If the source of a poll can't or won't tell you how they sampled respondents, how they interviewed them, what the questions and response options were, what the response rate was, or other details about the poll, then the results should be taken with a commensurate grain of salt," he advises.
Also, be skeptical of a poll if it was designed and conducted by someone without recognized credentials, experience and reputation, says Reineke. Just you'd be skeptical about a doctor without a degree or a journalist without any bylines.
Reineke suggests checking out the website of the American Association for Public Opinion Research (AAPOR). "If a pollster is not a member of AAPOR, or is dismissive of the organization -- or worse yet has never heard it of -- that should be cause for concern."
Poll Questions
One indicator of bias in surveys are leading questions, says Clark. For example: Are you better off now under the Obama administration than you were four years ago?
This question is biased because it ties Obama to the issue, says Clark.
"A poll is only as good as the questions asked," agrees Reineke. Questions should not encourage or discourage respondents to provide a particular response over others, and should only ask about one thing at a time.
Conversely, answers to questions should not include biased or politically charged words, says Clark. For example, phrases like "tax breaks for the rich" (instead of "tax reduction/reform"), "Obamacare" (rather than "healthcare reform") and "War on Terrorism" (instead of "War in Afghanistan") are all political labels with divisive meanings.
"Answers to questions that include these terms are more likely to be used by one party to validate their agendas," Clark explains. Thus, this is not a projectable measurement of public sentiment on issues.
Reineke also suggests considering these three guidelines regarding poll answers:
Response options should be exhaustive, meaning that any possible response is represented by a response option.
Response options should be mutually exclusive, meaning that participants will need one and only one response to indicate their answer.
Pollsters, and consumers of their results, should also pay attention to potential order effects, meaning the ways in which a previous question, or a participant's response to it, might affect interpretation or response to following questions.
Population Sampling
"Polls work by contacting a sample of the population of interest," says Reineke. That sample should be representative, meaning it should have the same proportion of all important characteristics as the population.
Representative samples are often achieved through random sampling, which means every member of the population has an equal chance of being selected, he says. "Pollsters should be prepared to explain how their sampling is random if they claim it is so."
"In cases where sampling is not random, pollsters should be able to explain how their sample is representative of the population, and provide appropriate cautions about the extension of results to groups who were not adequately represented in the sample," continues Reineke.
Population Size
"The size required for a random sample to be representative of the population in question is dependent on the size of the population," says Reineke. "The larger the sample, the smaller the margin of error."
In the simplest terms, "margin of error" is a statistic that shows how well the selected sample predicts things about the entire population.
Look at margins of errors when evaluating polls, suggests Schultz. "I would say any poll with margins of errors greater than +/- 4 are meaningless, since that means the results could be off by as much as eight points."
Interestingly, there is not much difference between the margin of error for a sample of 5,000 Americans vs. a sample of a million Americans, says Reineke. However, there is a significant difference in margin of error for a sample of 500 Americans vs. 2,500 Americans.
Statistical formulas aside, as a rule of thumb, you should look for a sample between 500 or 1,000 for state polls; and 1,000 or 2,000 for national polls, says Reineke.
"For presidential polls, I am suspect of any poll with survey samples of much less than 1,000 people," agrees Schultz. "They probably need about 1,200 to 1,500 people to be accurate, especially if one wants to tap into swing voters or the views of particular subgroups."
Also, ignore any poll that does not have a confidence level of at least 95 percent, says Schultz. Some polls have confidence levels of only 90 percent, which means they are only 90 percent confident that responses were within their margin of error. In other words, 10 percent of the time they are not sure if sample answers were indicative of the true population (not good).
Furthermore, polls are only as good as the underlying assumptions that go into them, continues Schultz. For example, a poll that lists 50 percent of those who responded as Democrats is skewed in terms of over-representing Democrats.
That's why samples are sometimes weighted to better represent the population of interest, says Reineke. For example, if African-American males ages 18-35 are 1 percent of the sample, but 2 percent of the population, a pollster might mathematically adjust the sample so that responses of individuals in that demographic actually count as two responses each, thus better reflecting the population.
Regardless, pollsters should report their sample size and their margin of error, and provide information about how they sampled so that others can evaluate their claims and methods, Reineke stresses.
Gracie
Dear Gracie,
With elections approaching, I've seen a lot of polls in the news recently. How do we know if the polls are accurate or biased?
Puzzled by Polls
Dear Puzzled by Polls,
Three ProfNet experts provide some insight: What You Need to Understand About Polls
"Creating and fielding a poll is not something that just anyone can do at the drop of a hat," says Jason Reineke, associate director of the Middle Tennessee State University (MTSU) Poll, which is a statewide, biannual poll of Tennesseans; as well as the university's assistant professor of journalism.
"It is both an art and a science, and the people who do it well usually have extensive training and expertise," continues Reineke. "Like a journalist, lawyer or medical doctor, being a pollster is a profession."
Polls are snapshots in time and not predictive tools, explains David Schultz, law and graduate school professor at Hamline University's School of Law, and editor of the Journal of Public Affairs Education. For example, polls conducted today about the presidential elections are not necessarily indicative of what will happen in November.
"A common problem with political polls is that they are often fielded by one party to support its agenda," adds Bob Clark, president of 24K Marketing.
Some polls are better than others, but the value of a poll can be better determined by the goals that it was designed to address, rather than one-size-fits-all rules, says Reineke. "Nonetheless, there are some standards that can be applied across most polls."
Transparency
Pollsters should freely and honestly report information about the poll's funding, affiliation, methodology, data and analysis, explains Reineke.
"If the source of a poll can't or won't tell you how they sampled respondents, how they interviewed them, what the questions and response options were, what the response rate was, or other details about the poll, then the results should be taken with a commensurate grain of salt," he advises.
Also, be skeptical of a poll if it was designed and conducted by someone without recognized credentials, experience and reputation, says Reineke. Just you'd be skeptical about a doctor without a degree or a journalist without any bylines.
Reineke suggests checking out the website of the American Association for Public Opinion Research (AAPOR). "If a pollster is not a member of AAPOR, or is dismissive of the organization -- or worse yet has never heard it of -- that should be cause for concern."
Poll Questions
One indicator of bias in surveys are leading questions, says Clark. For example: Are you better off now under the Obama administration than you were four years ago?
This question is biased because it ties Obama to the issue, says Clark.
"A poll is only as good as the questions asked," agrees Reineke. Questions should not encourage or discourage respondents to provide a particular response over others, and should only ask about one thing at a time.
Conversely, answers to questions should not include biased or politically charged words, says Clark. For example, phrases like "tax breaks for the rich" (instead of "tax reduction/reform"), "Obamacare" (rather than "healthcare reform") and "War on Terrorism" (instead of "War in Afghanistan") are all political labels with divisive meanings.
"Answers to questions that include these terms are more likely to be used by one party to validate their agendas," Clark explains. Thus, this is not a projectable measurement of public sentiment on issues.
Reineke also suggests considering these three guidelines regarding poll answers:
Response options should be exhaustive, meaning that any possible response is represented by a response option.
Response options should be mutually exclusive, meaning that participants will need one and only one response to indicate their answer.
Pollsters, and consumers of their results, should also pay attention to potential order effects, meaning the ways in which a previous question, or a participant's response to it, might affect interpretation or response to following questions.
Population Sampling
"Polls work by contacting a sample of the population of interest," says Reineke. That sample should be representative, meaning it should have the same proportion of all important characteristics as the population.
Representative samples are often achieved through random sampling, which means every member of the population has an equal chance of being selected, he says. "Pollsters should be prepared to explain how their sampling is random if they claim it is so."
"In cases where sampling is not random, pollsters should be able to explain how their sample is representative of the population, and provide appropriate cautions about the extension of results to groups who were not adequately represented in the sample," continues Reineke.
Population Size
"The size required for a random sample to be representative of the population in question is dependent on the size of the population," says Reineke. "The larger the sample, the smaller the margin of error."
In the simplest terms, "margin of error" is a statistic that shows how well the selected sample predicts things about the entire population.
Look at margins of errors when evaluating polls, suggests Schultz. "I would say any poll with margins of errors greater than +/- 4 are meaningless, since that means the results could be off by as much as eight points."
Interestingly, there is not much difference between the margin of error for a sample of 5,000 Americans vs. a sample of a million Americans, says Reineke. However, there is a significant difference in margin of error for a sample of 500 Americans vs. 2,500 Americans.
Statistical formulas aside, as a rule of thumb, you should look for a sample between 500 or 1,000 for state polls; and 1,000 or 2,000 for national polls, says Reineke.
"For presidential polls, I am suspect of any poll with survey samples of much less than 1,000 people," agrees Schultz. "They probably need about 1,200 to 1,500 people to be accurate, especially if one wants to tap into swing voters or the views of particular subgroups."
Also, ignore any poll that does not have a confidence level of at least 95 percent, says Schultz. Some polls have confidence levels of only 90 percent, which means they are only 90 percent confident that responses were within their margin of error. In other words, 10 percent of the time they are not sure if sample answers were indicative of the true population (not good).
Furthermore, polls are only as good as the underlying assumptions that go into them, continues Schultz. For example, a poll that lists 50 percent of those who responded as Democrats is skewed in terms of over-representing Democrats.
That's why samples are sometimes weighted to better represent the population of interest, says Reineke. For example, if African-American males ages 18-35 are 1 percent of the sample, but 2 percent of the population, a pollster might mathematically adjust the sample so that responses of individuals in that demographic actually count as two responses each, thus better reflecting the population.
Regardless, pollsters should report their sample size and their margin of error, and provide information about how they sampled so that others can evaluate their claims and methods, Reineke stresses.
Gracie
Subscribe to:
Posts (Atom)