Monday, November 28, 2011
How did the pollsters go?
A couple of weeks ago I wrote a post comparing the various polling companies that work in New Zealand, trying to see if there were any systematic biases in the results each company produces. In that post, it wasn't possible to compare each company's estimates to an external standard because, without an election its impossible to know exactly what the population is thinking. On Saturday night we got to know just what the electorate was thinking, and there were a couple of surprises.
So, here's the last estimate of each polling company compared with the results from Saturday night for each of the four parties that made the 5% cut:
The solid line is the election result for each party. Dashed lines are a 95% margin of error for a poll with 1 000 respondents and the given level of support
As you can see, the polls had National too high and New Zealand First too low more or less across the board. It's hard to say any company did better than the others, Roy Morgan got New Zealand First about right and were the closest to National's final result, but they made up for that by over-estimating the Green Party and under-estimating Labour.
So, why did most of the companies get New Zealand First and National wrong? Of course, it's possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn't vote. So, some of the people who recorded a preference in a poll didn't bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don't ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.
It's not possible to reconstruct the motivations of those people that didn't show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a 'likely voter' question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It's not perfect, but it's all we've got, and it doesn't really fit the story. National actually had stronger support in electorates with high turnout and there isn't much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that's not near to significantly different from no increase):
So, why did most of the companies get New Zealand First and National wrong? Of course, it's possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn't vote. So, some of the people who recorded a preference in a poll didn't bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don't ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.
It's not possible to reconstruct the motivations of those people that didn't show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a 'likely voter' question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It's not perfect, but it's all we've got, and it doesn't really fit the story. National actually had stronger support in electorates with high turnout and there isn't much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that's not near to significantly different from no increase):
Percentage turnout is calculated relative to total enrollment for electorate (special votes not yet included)
By the way, that graph, with such poor turnout in the electorates that with the largest proportion of Labour voters, might be depressing viewing for Labour supporters. Indeed, I'm sure they'll want to work on "getting out the vote" in the Māori electorates and South Auckland, but that pattern doesn't explain the erosion of Labour's support in this election. Turnout was low in these electorates in 2008. Here's the support for each party plotted against change in turnout for each electorate (not including special votes):
Labels: election, pretty-data, sci-blogs
Saturday, November 26, 2011
Will early returns accurately predict tonight's outcome?
It's election day.
I'm sure that later this evening we'll be treated to one of the staples of election night coverage, results from early returns delivered with the qualifier that they are unlikely to represent a reliable estimate of the final result.
Part of that reason for that qualifier is the obvious point that any small sample only provides an uncertain estimate what's going on in the population large. That sampling error can be compounded if there is some bias that makes the early-returning booths pull further left or right than the population at large. In New Zealand at least, you'll often here people say the small polling booths that quickly count and report their results are predominantly from rural areas and, so, favor right wing parties compared with larger, urban booths. (The meme that passes among some right-wing folks, that Labour only won the 2005 election because of South Auckland voters is an example of this narrative).
So, I thought I'd see if there was any evidence for that these stories are right. It's easy to get the data from each of the 6 000 or so polling booths used in the 2008 election and munge it together to see if there is a relationship between how red or blue a booth was and how many people voted there (which is a pretty good proxy for how quickly the results will be reported):
I'm sure that later this evening we'll be treated to one of the staples of election night coverage, results from early returns delivered with the qualifier that they are unlikely to represent a reliable estimate of the final result.
Part of that reason for that qualifier is the obvious point that any small sample only provides an uncertain estimate what's going on in the population large. That sampling error can be compounded if there is some bias that makes the early-returning booths pull further left or right than the population at large. In New Zealand at least, you'll often here people say the small polling booths that quickly count and report their results are predominantly from rural areas and, so, favor right wing parties compared with larger, urban booths. (The meme that passes among some right-wing folks, that Labour only won the 2005 election because of South Auckland voters is an example of this narrative).
So, I thought I'd see if there was any evidence for that these stories are right. It's easy to get the data from each of the 6 000 or so polling booths used in the 2008 election and munge it together to see if there is a relationship between how red or blue a booth was and how many people voted there (which is a pretty good proxy for how quickly the results will be reported):
If polling booths were unbiased samples of the electorate (they really aren't...), we'd expect all 99% of the dots in these graphs to fall between those dashed lines.
As you can see, at least in 2008, the common wisdom appears to be wrong: the larger polling booths went more toward National and less toward Labour than the rest of the country.
Tonight's early results will certainly only be an indicator of where things will end up, but things won't necessary get better for left-leaning parties as the night goes on.
UPDATE Thanks for a comment from Luis Apiolaza, who suggests a better way of getting the question. These plots show the percent favouring each major party after each polling booth's data is added in order of how many votes were recorded. As you can see, it takes a long time for Natinoal's vote to approach its final number. I wonder if this is in part be an artifact of many small booths in Maori electorates? Of course, polling booths won't necessarily report in order of how many votes were cast in them.
UPDATE Thanks for a comment from Luis Apiolaza, who suggests a better way of getting the question. These plots show the percent favouring each major party after each polling booth's data is added in order of how many votes were recorded. As you can see, it takes a long time for Natinoal's vote to approach its final number. I wonder if this is in part be an artifact of many small booths in Maori electorates? Of course, polling booths won't necessarily report in order of how many votes were cast in them.
Wednesday, November 9, 2011
Polling bias among local pollsters?
Lew from Kiwipolitco stumbled across an interesting analysis of Australian pollsters, looking at whether the methods employed by particular polling companies might lead to biases in the numbers they produce .
As that post notes, it's impossible to tell whether any particular pollster is showing a bias with respect to the real levels of support for each party, because we only find out what the whole population is thinking when we have an election. We can, however, see if a particular polling company is pulling in a different direction than others. Since I've had about all the thesis-editing I can stand for one day, I decided to see if the data I put together for those charts from this morning could tell us anything about polling bias among New Zealand pollsters.
Obviously, the first step is decide on how to find the "average" number against which all polls should be compared. I went with a local regression which just fits a smoothed line through all the points. From there you can measure how far a single estimate is from the smoothed value at the same date*:
Even if all the companies are perfectly sampling the population, we'd expect estimates from single polls to show a good deal of scatter, just because polls are estimates of the population value and come with uncertainty. Indeed, there are dots all over the place in that graph. By collating all those differences between a point and the smoothed lines for the same date we can see if any of the companies are consistently finding higher or lower estimates of a party's support:
And the answer is... sort of. It seems TV3 tends to get National a little higher and Labour a little lower than the rest of the pollsters, and perhaps the Herald goes the other way. I think trying to gain anything more meaningful than that from these results is probably the statistical equivalent of reading tea leaves, but feel free to stare at the graph and confirm your own political biases!
*You should note, this approach introduces biases of its own and you should take these results with several grains of salt. In particular Roy Morgan polls a lot more often than the other companies, so that company's polling will contribute more towards the smoothed line than others, so it's no surprise they show only small biases.
If you are interested in trends in polling bias, I have these differences plotted over time here. As always, I have the raw data here so let me know if you want a csv file to play with.
Obviously, the first step is decide on how to find the "average" number against which all polls should be compared. I went with a local regression which just fits a smoothed line through all the points. From there you can measure how far a single estimate is from the smoothed value at the same date*:
And the answer is... sort of. It seems TV3 tends to get National a little higher and Labour a little lower than the rest of the pollsters, and perhaps the Herald goes the other way. I think trying to gain anything more meaningful than that from these results is probably the statistical equivalent of reading tea leaves, but feel free to stare at the graph and confirm your own political biases!
*You should note, this approach introduces biases of its own and you should take these results with several grains of salt. In particular Roy Morgan polls a lot more often than the other companies, so that company's polling will contribute more towards the smoothed line than others, so it's no surprise they show only small biases.
If you are interested in trends in polling bias, I have these differences plotted over time here. As always, I have the raw data here so let me know if you want a csv file to play with.
Labels: election, pretty-data, sci-blogs
All content not otherwise marked is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
