How did the pollsters go?

A couple of weeks ago I wrote a post comparing the various polling companies that work in New Zealand, trying to see if there were any systematic biases in the results each company produces. In that post, it wasn't possible to compare each company's estimates to an external standard because, without an election its impossible to know exactly what the population is thinking. On Saturday night we got to know just what the electorate was thinking, and there were a couple of surprises. 

So, here's the last estimate of each polling company compared with the results from Saturday night for each of the four parties that made the 5% cut:

 

The solid line is the election result for each party. Dashed lines are a 95% margin of error for a poll with 1 000 respondents and the given level of support

As you can see, the polls had National too high and New Zealand First too low more or less across the board. It's hard to say any company did better than the others, Roy Morgan got New Zealand First about right and were the closest to National's final result, but they made up for that by over-estimating the Green Party and under-estimating Labour.

So, why did most of the companies get New Zealand First and National wrong? Of course, it's possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn't vote. So, some of the people who recorded a preference in a poll didn't bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don't ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.

It's not possible to reconstruct the motivations of those people that didn't show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a 'likely voter' question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It's not perfect, but it's all we've got, and it doesn't really fit the story. National actually had stronger support in electorates with high turnout and there isn't much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that's not near to significantly different from no increase):

Percentage turnout is calculated relative to total enrollment for electorate (special votes not yet included)

By the way, that graph, with such poor turnout in the electorates that with the largest proportion of Labour voters, might be depressing viewing for Labour supporters. Indeed, I'm sure they'll want to work on "getting out the vote" in the Māori electorates and South Auckland, but that pattern doesn't explain the erosion of Labour's support in this election. Turnout was low in these electorates in 2008. Here's the support for each party plotted against change in turnout for each electorate (not including special votes):

All content not otherwise marked is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.