Monday, November 28, 2011
How did the pollsters go?
A couple of weeks ago I wrote a post comparing the various polling companies that work in New Zealand, trying to see if there were any systematic biases in the results each company produces. In that post, it wasn't possible to compare each company's estimates to an external standard because, without an election its impossible to know exactly what the population is thinking. On Saturday night we got to know just what the electorate was thinking, and there were a couple of surprises.
So, here's the last estimate of each polling company compared with the results from Saturday night for each of the four parties that made the 5% cut:
The solid line is the election result for each party. Dashed lines are a 95% margin of error for a poll with 1 000 respondents and the given level of support
As you can see, the polls had National too high and New Zealand First too low more or less across the board. It's hard to say any company did better than the others, Roy Morgan got New Zealand First about right and were the closest to National's final result, but they made up for that by over-estimating the Green Party and under-estimating Labour.
So, why did most of the companies get New Zealand First and National wrong? Of course, it's possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn't vote. So, some of the people who recorded a preference in a poll didn't bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don't ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.
It's not possible to reconstruct the motivations of those people that didn't show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a 'likely voter' question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It's not perfect, but it's all we've got, and it doesn't really fit the story. National actually had stronger support in electorates with high turnout and there isn't much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that's not near to significantly different from no increase):
So, why did most of the companies get New Zealand First and National wrong? Of course, it's possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn't vote. So, some of the people who recorded a preference in a poll didn't bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don't ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.
It's not possible to reconstruct the motivations of those people that didn't show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a 'likely voter' question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It's not perfect, but it's all we've got, and it doesn't really fit the story. National actually had stronger support in electorates with high turnout and there isn't much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that's not near to significantly different from no increase):
Percentage turnout is calculated relative to total enrollment for electorate (special votes not yet included)
By the way, that graph, with such poor turnout in the electorates that with the largest proportion of Labour voters, might be depressing viewing for Labour supporters. Indeed, I'm sure they'll want to work on "getting out the vote" in the Māori electorates and South Auckland, but that pattern doesn't explain the erosion of Labour's support in this election. Turnout was low in these electorates in 2008. Here's the support for each party plotted against change in turnout for each electorate (not including special votes):
Labels: election, pretty-data, sci-blogs
Wednesday, November 9, 2011
Polling bias among local pollsters?
Lew from Kiwipolitco stumbled across an interesting analysis of Australian pollsters, looking at whether the methods employed by particular polling companies might lead to biases in the numbers they produce .
As that post notes, it's impossible to tell whether any particular pollster is showing a bias with respect to the real levels of support for each party, because we only find out what the whole population is thinking when we have an election. We can, however, see if a particular polling company is pulling in a different direction than others. Since I've had about all the thesis-editing I can stand for one day, I decided to see if the data I put together for those charts from this morning could tell us anything about polling bias among New Zealand pollsters.
Obviously, the first step is decide on how to find the "average" number against which all polls should be compared. I went with a local regression which just fits a smoothed line through all the points. From there you can measure how far a single estimate is from the smoothed value at the same date*:
Even if all the companies are perfectly sampling the population, we'd expect estimates from single polls to show a good deal of scatter, just because polls are estimates of the population value and come with uncertainty. Indeed, there are dots all over the place in that graph. By collating all those differences between a point and the smoothed lines for the same date we can see if any of the companies are consistently finding higher or lower estimates of a party's support:
And the answer is... sort of. It seems TV3 tends to get National a little higher and Labour a little lower than the rest of the pollsters, and perhaps the Herald goes the other way. I think trying to gain anything more meaningful than that from these results is probably the statistical equivalent of reading tea leaves, but feel free to stare at the graph and confirm your own political biases!
*You should note, this approach introduces biases of its own and you should take these results with several grains of salt. In particular Roy Morgan polls a lot more often than the other companies, so that company's polling will contribute more towards the smoothed line than others, so it's no surprise they show only small biases.
If you are interested in trends in polling bias, I have these differences plotted over time here. As always, I have the raw data here so let me know if you want a csv file to play with.
Obviously, the first step is decide on how to find the "average" number against which all polls should be compared. I went with a local regression which just fits a smoothed line through all the points. From there you can measure how far a single estimate is from the smoothed value at the same date*:
And the answer is... sort of. It seems TV3 tends to get National a little higher and Labour a little lower than the rest of the pollsters, and perhaps the Herald goes the other way. I think trying to gain anything more meaningful than that from these results is probably the statistical equivalent of reading tea leaves, but feel free to stare at the graph and confirm your own political biases!
*You should note, this approach introduces biases of its own and you should take these results with several grains of salt. In particular Roy Morgan polls a lot more often than the other companies, so that company's polling will contribute more towards the smoothed line than others, so it's no surprise they show only small biases.
If you are interested in trends in polling bias, I have these differences plotted over time here. As always, I have the raw data here so let me know if you want a csv file to play with.
Labels: election, pretty-data, sci-blogs
Election graphics
As you may have noticed, there is an election around the corner. I'm not about to subject anyone to my political opinions, but I do have an area interest in which I could probably match the most tribal Labour or National supporter for fervent opinion. Bad graphs. Being able to clearly and succinctly display data is a really important scientific skill, and it's one that some of our newspapers have so far failed to develop. Take this chart, which appeared in the Christchurch Press and is apparently showing the break-down of a recent poll (thanks to John Kerr for pointing this out on Stats Chat)
*It should be pointed out, these thresholds are really indicative rather than absolute. The real cuts with vary depending on how many votes go to parties that don't make the 5% threshold or win an electorate seat and the size of any overhang
The polling data used here is taken from Wikipedia. Contact me if you want a csv file with the dates in an R-friendly format. The stacked bar charts were done manually in Inkscape and all the other plots were done with ggplot then tweaked in Inkscape.
I have no words. I honestly have no idea why there is a third dimension in this chart, or, really what I'm meant to take from any of it. This was so bad I had to have a go at fixing it.
The point of a political poll is to tell which party, along with its idealogical allies, is likely to win a majority of the seats in parliament. Since we are talking about proportions, and there is a real "something" that those proportions add up to, this a rare case in which a pie chart (or its bastard cousin the torus) might make an effective visualisation. Even so, people are generally better and assessing differences between lengths of objects than their area, so I went with a stacked bar chart.
Of course, even MMP isn't a perfectly proportional electoral system, and, especially with the overhang likely caused by the Māori Party winning more electorate seats that their party votes alone would entitle them to, the break-down of seats in parliament won't exactly match the break-down of the party vote. Here's what would happen if these results were mirrored on election day, and Act, Mana and the Māori Party held their electorate seats while the fine people of Ohariu-Belmont remove Peter Dunne, the Member for the anti-1080 lobby.
In this case, if Act were to lose Epsom the next list seat would go to the Green Party. Of course, you shouldn't use a single poll to say anything very meaningful about what a specific result is likely to be. Polls are estimates of what's going on in the population at large, and estimates always come with uncertainty. We can go some way to quantifying that uncertainty but giving each estimate a "margin of error". Using some slightly twisty logic you can say, "we don't know what the support for the National Party really is, but if it was 52.6 %, as our poll suggests, and we did lots of surveys, each of 1000 people, then 95% of the time we'd get a result between 49.5% and 55.7%". It's hard to include that uncertainty in a stacked bar chart, so here's each estimate from that poll, along with its margin of error (the "2nd seat" line is at 1.2%, the point at which a party with an electorate seat would probably get another on the list, the "govern alone" line is 51.6%, the proportion of party votes required to get 62 seats, a majority in a 123 seat parliament*)
If the fairfax poll was the only thing we had to go on, it might give left-leaning folks (like me) some hope that the true value of the National Party support was less than the "govern alone" threshold. Unfortunately, we have a whole series of polls conducted over the course of the year that suggest National's support is pretty close to the point estimate, and there appears to be little trend across those polls (except perhaps the Green party picking up Labour supporters):
That chart munges all the political polls taken this year together, and fits a smooth line between them. But some polls are probably better than others, it's hard to take a truly representative sample of the country, and it's likely that some of the companies running the polls do a better job. So here's the trend across each polling company:
And you can do the same thing looking at each of the smaller parties.
I'm sure there are lots of ways you could improve these charts (let me know if you think of some), but surely they are all more useful than that awful one from The Press?
*It should be pointed out, these thresholds are really indicative rather than absolute. The real cuts with vary depending on how many votes go to parties that don't make the 5% threshold or win an electorate seat and the size of any overhang
The polling data used here is taken from Wikipedia. Contact me if you want a csv file with the dates in an R-friendly format. The stacked bar charts were done manually in Inkscape and all the other plots were done with ggplot then tweaked in Inkscape.
Labels: pretty-data, sci-blogs
All content not otherwise marked is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
