Polling, simplified: What to keep in mind when reading election polls

Is polling broken? This is a question that has been asked since the 2016 presidential election, when Donald Trump’s victory over Hillary Clinton left many Americans wondering whether polls can accurately capture public opinion and predict election outcomes. Since the 2016 presidential election there have been other polling misses, most recently in the 2021 gubernatorial races in Virginia and New Jersey. Patrick Murray, the director of Monmouth University Poll in New Jersey, went so far as to apologize for the inaccuracy of the polls and question the utility of election polls going forward.

While Murray may be flirting with the idea of doing away with election polls, it is unlikely that they will disappear from our political and media ecosystems in any meaningful way in the near future. Because of this, it is important for all citizens, from the average voter to the seasoned campaign operative, to have a strong understanding of what exactly a poll is saying and what should be considered when reading a poll. Public Wise is here to break down the top things to remember when looking at a poll, whether you’re thumbing through cross tabs on a regular basis or skimming articles that utilize polling in your local paper.

We’ve divided our tips and tricks into two different categories. The first is what to keep in mind when reading a specific poll. Say your preferred news source or local pollster just released a new poll, what should be top of mind when you open the article or memo for the first time?

Things to consider when reading a specific poll:

1. Who conducted the poll? Pollsters can be wrong and some are better than others. FiveThirtyEight maintains pollster ratings, which are based on historical polling accuracy and adjusted for things like the type of election, sample size, and the performance of other polls. These ratings can be a useful tool when assessing the quality of sources. Additionally, pollsters can be biased! Is it a public pollster? An internal campaign poll? A news source? Looking at who conducted the poll may give you clues as to who the poll might be inclined to favor (or oppose). However, keep in mind that who fielded a poll (and their partisan bias, if they have any) is not always immediately apparent because many polling firms conduct polls for other groups. Moreover, even when a pollster does not have a partisan bias, their polling can still be consistently inaccurate.
2. When was the poll conducted? Remember, polls are snapshots in time. Always look at when a poll was fielded and consider major events that happened just before the poll was fielded that may have impacted the results. Also consider what has occurred since the poll came out of the field and how that could shift public opinion. For example, the Supreme Court’s decision that overturned Roe v Wade will undoubtedly affect the results of the 2022 midterms, but the extent to which it will affect the election remain unclear even as abortion access remains a top issue in polling.
3. How was the poll conducted? Did the poll call voters at home or on their cell phones? Did it use online questionnaires? Did it use a mix of methods? Live caller polling had long been the gold standard, but as internet access expanded so did the use of online polls. The method of polling matters and the effect of a given method is something that is constantly evolving. For example, in 2012 FiveThirtyEight found that online polls performed slightly better than telephone polling overall. However, in 2016, they found that Hillary Clinton’s lead was larger in live interview polls compared to non-live polls. Of course, the method only matters insomuch as it affects the representativeness of the sample and we do not yet have a clear understanding of the quality of online samples as they compare to phone based sampling. Moreover, sample quality can vary from pollster to pollster based on many different factors that may or may not be related to polling method.
4. Who was polled? A given poll typically looks at the general population, registered voters, or likely voters. The group polled can have a significant impact on results (and making comparisons between polls with different target samples can be very tricky). Voters are generally older, whiter, and thus more Republican and about 40% of Americans are not registered to vote. Thus, deciding to poll only likely voters or registered voters restricts the sample in a way that can skew results.

The second group of tips are things to remember in any given election cycle, when you will surely be flooded with polls and surveys for months on end. This advice focuses on the big picture and things that can impact the polling ecosystem at large.

Things to consider when reading polls more generally:

1. Focus on frequency. When polling results and election models are communicated and visualized, they often emphasize the probability of a given outcome, such as a given candidate winning an election or a party taking/maintaining control. This has a few negative side effects, including that probabilities typically do not properly convey uncertainty, or how different estimates from the sample are from the true values in the population of interest. This is something Northwestern Computer Scientist Jessica Hullman discusses at length here. Moving beyond the wording or visualization of the source and imagining how else the results could be represented can help readers reach beyond this probability based outlook. For presidential elections, for example, it can be useful to focus on the frequency of outcomes in repeated simulations, which helps convey not only probabilities but how those probabilities are determined. Northwestern’s Matt Kay did that well here with a presidential plinko board.
2. Look at a variety of polls from a variety of sources. A sample of one is not a particularly helpful sample, both when conducting a poll and when reading them. Looking at a large selection of polls from a number of different sources will give readers a better understanding of the landscape as a whole. Pollsters make different decisions when conducting a poll and interpreting results, as the New York Times demonstrated when they gave four pollsters the same raw data and wound up with 4 different sets of results.
3. Be wary of pollster herding. Related to the previous point, pollsters are often disinclined to publish polls that deviate too much from what other pollsters are putting out. Because of this, pollsters might hold outlier polls. But sometimes, outlier polls are not outliers at all, they are accurate. That was the case in the 2014 Virginia Senate race, when outlier pollsters decided not to publish findings in the final stretch of the campaign. Conversely, in 2020 Ann Selzer of the Iowa Poll released results in the run up to the general election that put President Donald Trump 7 points up in Iowa. These results were a departure from the tighter race polls had been predicting in Iowa and Selzer faced a great deal of blowback for releasing what was considered an “outlier” poll. When all was said and done, Seltzer was right. Trump won Iowa by 8 points.
4. Remember to look at the margin of error. The margin of error reported by a pollster accounts for the variability around a candidate’s support level. Pew Research Center does a great job explaining margin of error and its role when interpreting results. Most importantly, they explain interpreting margin of error as follows: “A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.”
5. Framing matters. Do not just look at the results when reading a poll, look at how the questions were asked. Earlier this year, Public Wise conducted a poll where respondents were asked about their thoughts on President Joe Biden’s decision to nominate a Black woman to the Supreme Court. The survey found, among other things, that how you phrase a question and the context you do (or do not) provide can have an enormous impact on responses. Always look at how a question is asked when interpreting results!

There is a never ending laundry list of things to consider when reading an election poll. Whether the poll is looking at public opinion in a single state or is being used as part of a larger model to predict a presidential election outcome, it is easy to lose sight of the forest for the trees. Polling is a helpful tool, but in recent years the calibration has been off. Public Wise’s Dr. Jessica Kalbfeld breaks down some of the recent problems with polls and why we shouldn’t always rely on them so heavily in the electoral context. But in the meantime, we hope these tips will help you make the most of polls this primary and general election season.