Decision Mechanics

Insight. Applied.

Is wrong better than nothing?

By

Last week Italians voted “No” to constitutional reform. The Economist reports that

The 20-point margin of defeat—–60% to 40%—–was almost double what pollsters had foreseen.

Yet again, polling has failed. However, come the French and German elections next year who doubts that we’ll continue to celebrate or denounce each poll result with partisan fervor?

There’s a hunger for this kind of information and consumers will take something over nothing—even if that something isn’t telling them very much.

I recently wrote about how the Idiosyncratic Rater Effect renders a lot of human resource data next to useless—yet organizations continue to collect and act on it.

If we want things to change it seems we can’t just point out that the emperor has no clothes—we have to dress him.

RIP polling

By

voting intention survey

Polling died last night.

It’s been terminally ill for a while now. The predictions for the UK general election in 2015 were abysmal. Brexit polls were unreliable. And polls put the Scottish independence referendum result in 2014 as a close call when it was a resounding “No”.

After its performance in last night’s US presidential election, we have to reach for the life support switch.

Here are the poll results immediately prior to the election.

PollWinnerMargin
Monmouth UniversityClinton+6
Lucid/The Times-PicayuneClinton+5
ABC News/Washington PostClinton+4
Fox NewsClinton+4
Insights WestClinton+4
New York Times/CBS NewsClinton+4
YouGov/EconomistClinton+4
Bloomberg/SelzerClinton+3
RasmussenClinton+2
IBD/TIPPTrump+2

FiveThirtyEight, which became famous for its predictions in previous US elections, had Clinton with a 71.4% chance of winning over Trump’s 28.6%. It also had Clinton comfortably winning the popular vote by 48.5% against 44.9%.

The Independent reported on a model from Moody’s Analytics that has correctly picked every president since 1980. Well…had. It forecast Clinton would pick up 332 Electoral College seats to Trump’s 206.

Sam Wang of the Princeton Election Consortium called the result on 18 October—a win for Clinton. He tweeted

It is totally over. If Trump wins more than 240 electoral votes, I will eat a bug.

That’s lunch today sorted then.

So much time, energy and column inches have been spent on techniques which, again and again, have come up short. We can still have these polls, but we need to get them off the front page and make space for them under the horoscopes.

The problem is that we all desperately want to know what is going to happen. So, there’s a market for people who profess to be able to tell us. Given the demand, if we are going to euthanize polling, we need a replacement.

Prediction markets seemed promising. However, when I looked at Betfair Predicts before the election it was giving Clinton an 83% chance of success. An average of nine predication markets (including Betfair) published just before the election gave Clinton a 82.5% chance of victory.

So much for that then.

Polling (and betting) is based on obtaining people’s opinions—ideally a lot of people’s opinions. Unfortunately, when we lack any reasonable precedent for a situation or decision, it’s very hard to have any kind of informed opinion. Becoming informed about complex socio-economic situations takes resources—an investment very few are willing to make just to enhance the accuracy of a one-off prediction of an event they can’t change.

Crowds have no wisdom when the individuals don’t have a clue.

What about those who put a bit more time into their opinions? Well, the superforcasters at the Good Judgement Project reported a 76% chance that a Democrat would win and a 64% chance that the Democrats would control the Senate.

So much for that then.

Is there anything we can do to predict the outcomes of these elections?

As physicist and Nobel laureate Neil Bohr said

Prediction is very difficult, especially about the future.

If we can’t rely on judgement the only way forward would seem to be to improve our ability to model and study the social physics of complex systems—such as the research published in the Journal of Artificial Societies and Simulation. We’re currently a long way from being able to use such approaches with any degree of confidence, but the techniques used in this field, such as agent-based simulation, have the potential to make predictions in novel situations.

Such techniques also tend to be expensive to use—especially when compared with running an online survey. However, we can’t just keep doing things that clearly aren’t working just because we can.

RIP polling. I won’t mourn you.

Don’t trust the polls

By

2016 is the year of the US presidential election. Prepare to be besieged by polls. The embarrassment of the 2015 UK parliamentary election predictions is a distant memory. We get to start over.

However, Mona Chalabi reminds us, via the Guardian’s Datablog, of the challenges facing pollsters. She lists five:

  • The media are fallible. They follow fashion and make the news.
  • Journalists are fallible. They are biased—just like the rest of us.
  • Predicting the future is hard. Context shifts.
  • It’s difficult to reach people. There’s no longer a household landline.
  • People don’t want to be polled. The few who do probably have something in common.

That’s a pretty damning list—and it’s far from exhaustive.

Of course, if we don’t have poll results we’ll have to have news stories about the issues…and who’s got time for that?

Election polling errors blamed on bias

By

UK polling station sign

A report has concluded that the spectacular failure of pollsters to predict the result of the 2015 UK Parliamentary elections was largely due to

systematic over-representation of Labour voters and under-representation of Conservative voters

The report, compiled by a panel of academics and statisticians, was commissioned by the polling industry to determine why they had predicted a “photo-finish” in an election where Conservatives outpolled Labour by 36.9% to 30.4%—a crushing defeat for Labour that lead to the resignation of their leader.

Pollsters apparently used collection methods that were more likely to be used by young (Labour-leaning) voters than older (Conservative-leaning) voters. Frankly, not realising that online surveys are going to under-represent the over 70s is a shocking oversight.

While betting markets also under-estimated the extent of the win, they did better than the polling industry—without the expense.