Asking
Only logged in members can reply and interact with the post.
Join SimilarWorlds for FREE »

Pollsters were wrong. AGAIN. Please explain why.




I can only think of two reasons why a predicted Kamala grand slam instead became a hat trick for Republicans (they now control the white house, senate, and house of representatives).

First possibility : millions of people lied to pollsters.

2nd possibility : those Pollsters lied to us.

Which do you think it was?
This page is a permanent link to the reply below and its nested replies. See all post replies »
Nibblesnarf · 26-30, M
As the results finalize, it looks like swing state polling may have been a bit better than it was the last two presidential elections. But the familiar trend did emerge: polls underestimated Trump's performance almost across the board. At least some of the misses in polling averages were too much to credibly explain via random sampling error.

It should be noted that a few pollsters did very well. I'll give a shoutout to AtlasIntel, which polled the top battleground states with a freaky level of accuracy. (I'll also give a different kind of shoutout to Ann Selzer, whose final Iowa poll was a catastrophic miss.)

Now, to your question: polling is less of a science than many people believe, and there are numerous ways that things can go wrong. Here are a few.

1. Typically, pollsters don't publish their raw data. Once they've gathered their data, most will perform calculations that artificially make the demographics of their sample match the expected electorate. To put it another way, pollsters' assumptions are baked directly into their final published numbers. Done properly, this usually leads to better results. But it also makes it easy for bias to seep in (whether it be intentional or unintentional). Even an unbiased pollster can go astray if their assumptions about the electorate are wrong. I suspect (but don't know for sure) that this was part of the problem.

2. Poll "herding" seems to be a real phenomenon. Statistically speaking, outlier polls should happen sometimes. But nobody wants to actually be the outlier. So what does a pollster do if the election is near, and their result is far off of the polling average? Well, it seems that some pollsters in this situation muck with their numbers to make their result closer to the polling average - that is, closer to the "herd". The herding in the final few weeks was conspicuous enough that it was visible to the naked eye. Herding can cause the published polls to miss real shifts in the closing weeks. My guess is that herding didn't really create error this time. But it still means that some polls in the last few weeks weren't really giving us new information.

3. As Becky mentioned, it seems like Trump's voter base is just less likely on average to participate in polls. Pollsters seem to believe this was their main problem in 2020, and I think it likely accounted for part of the underestimation of Trump in 2024. It'll be interesting to see how polling does in the future, when Trump is no longer on the ballot.

4. When you see an election forecast, it's usually based on some method of aggregating polls together. This means that the forecast you see (e.g. "98% chance of a Hillary Clinton win") has already passed through two "filters": that of the pollster, and that of the aggregator. Averaging polls together is great because it should largely smooth out the effects of random sampling error. But the aggregator's modeling decisions and biases may have significant sway over their results. They might exclude or give lesser weight to pollsters they consider to be "low quality". (For example: fivethirtyeight chose to exclude Rasmussen from their 2024 model, for... somewhat questionable reasons, IIRC. And with the results in, Rasmussen appears to have been one of the most accurate pollsters of this election.)
SusanInFlorida · 31-35, F
@Nibblesnarf concealing "raw data" is like going to court and refusing to submit evidence. it's shady right from the get-go
Nibblesnarf · 26-30, M
@SusanInFlorida There was an experimental pollster in 2016 that published absolutely everything: their raw data, their weights and demographic assumptions, and details about their methodology. An interested reader could use this information to reproduce their published results. Unfortunately, this didn't become the norm in the industry.

Maybe pollsters keep some of this stuff close to the chest as a kind of "secret recipe", by which they hope to keep an edge over their competitors? I agree that it's shady. Poll results are reported with an air of scientific objectivity behind them, but in reality, they're often as much a product of pollster opinions/assumptions as they are of real world data.

I also remember an interesting exercise from the 2016 election cycle: someone gave the same raw data to four different pollsters, and asked them to process it as usual. There was relatively wide variance in the results, even though all four "polls" came from the same data.
SusanInFlorida · 31-35, F
@Nibblesnarf i dont need to see the "secret recipe". but it would be nice to see the ingredients, and if leading questions were used by the pollsters to drive the results in a manner favorable to whoever is paying the bills.
Nibblesnarf · 26-30, M
@SusanInFlorida I think big name pollsters usually do publish the exact questions they asked, so at least that level of transparency is often available. And the concern about leading questions (or other types of biased design) is more than valid. I've been polled a few times, and I did observe some unscrupulous behaviors.

With the other key ingredients, readers/watchers are usually left in the dark. I think some pollsters at least list the voter traits/demographics they choose to weight, but that's still very partial information. And there can be a devil in the hidden details if the poll sample is abnormal on a trait the pollster decided not to weight.