The Senate races in Arkansas and North Carolina are both toss-ups, and the polls seem to be trending in a Republican direction. But can the polls be trusted?
Videos By Rare
In Arkansas, incumbent Democrat Mark Pryor was leading in most polls in the spring, by double digits in some cases. But his Republican opponent, Rep. Tom Cotton, has been ahead in six polls taken in the last four months.
In North Carolina, the trend is less dramatic, but it points in the same direction. Democratic Senator Kay Hagan led in three polls taken in June. State House Speaker Thom Tillis has led in three out of four polls taken in July and August.
That could easily be an illusory trend, indicating that the race in North Carolina is tied and results are swinging simply because of statitistical variation. But that wouldn’t fully explain the wider swing in Arkansas.
It’s perfectly plausible that Cotton has made genuine gains in recent months, but there’s a wrinkle. FiveThirtyEight.com’s Harry Enten pointed out this week that in recent months none of the polls in these and several other states have been what he calls “gold-standard polls” – that is, standard probability samples from pollsters who don’t work for a candidate (or outside group supporting a candidate), use live interviewers, and fully disclose their methodology.
Not all deviations from that standard are created equal.
Internal polls released by a campaign or outside group supporting a campaign can be discounted entirely, because they release their data selectively. Polls released this month by the Pryor campaign and the Democratic Senatorial Campaign Committee showed Pryor ahead – but for all we know, they showed Cotton ahead in numerous polls that they kept private, and only released outliers to rally their troops.
Polls that incorporate data collected online have a record that’s mixed at best, though more media organizations are relying on them (they’re cheaper!). There are a lot of tricks that pollsters have tried to find a representative sample by contacting people over the internet, but nothing that’s proven reliable enough to establish anything like an industry standard.
Automated polls – where a recording asks respondents to answer questions by pushing numbers – seemed to perform reasonably well a decade ago, but have probably become less trustworthy as more households have eschewed landlines. In most cases, unsolicited calls to cell phones are only legal with live operators.
Enten identifies a reason to distrust these nontraditional methods even more: They perform better in races where the gold-standard pollsters are also in the field.
That means that nontraditional pollsters may be weighting their samples to conform to the consensus in the final days of the race. If that’s the case, those pollsters’ records of accuracy in predicting elections overstates how much they can be trusted in the middle of the summer when traditional polls aren’t being released as often.
The most plausible interpretation of the data is that the race in North Carolina is more or less static, the race in Arkansas has moved slightly in Cotton’s direction, and both states remain toss-ups.
But polling data right now should be consumed as part of a high-sodium diet. Never hesitate to add a few grains of salt.