Ken Ring can't predict the weather

It's that time of year for academics in New Zealand. As soon as undergrad teaching finishes every department, organisation and society on campus decides to schedule some sort of meeting,conference or symposium - since everyone will have so much spare time. Between the three talks I'm going to end up giving this month, and the tonne of work I have to get done between them, I don't have much time to write here. So, partly inspired by people talking abut their months-old posts in Grant's piece on the length of time it can take to put a blog post together, I've decided to dive in The Atavism's "draft" folder and resurrect a few half-written posts. This one took about two months to write, and though I never quite got the copy right I do rather like the graph.

Ken Ring was on National Radio a couple of months ago, blathering on about his method of weather prediction. Ring thinks he can provide a forecast for any future date by looking at weather map that's about 18 years out of date That's how long it takes for the earth, the moon and the sun to cycle into the same relative positions in their orbits, and Ring thinks it's the moon that drives weather down here on earth. The 18 year old weather map will tell him what the atmosphere was doing last time everything lined up this way, and that will be enough to predict the next event. You probably agree that Ring's methods sound like lunacy, but Ring continually claimed in his time on air that his method had an 85 % accuracy rate. Reading Ring's website, you can see he is pretty generous when he estimates his own accuracy, like the Texas sharpshooter who shoots the side of a barn then paints a target around the bullet hole to show his prowess with a shooting iron, Ring uses any vaguely similar weather event to prop up the accuracy of his predictions. My particular favourite from that page is his prediction for 100 mm or rain in New South Wales, which was accurate, it's just that it arrived further West, two days later and was a only 20mm.

So, we shouldn't take Ring's self confidence too seriously and his method, which involves tricky maths and obscure terminology, is a prefect example of cargo cult science. But we don't have to stop there, Ring makes specific predictions that we can test. Just as the worst thing you can say about homeopathy is not that it's impossible for those dilutions to effect the human body, but that homeopathy has been shown to do nothing; the worst thing you could say about Ring's weather forecasting methods is that they don't work. I went digging for some of Ring's old claims and was thrilled to see that my Sciblogs stablemate Gareth Renowden has already done all the hard work for me! Apparently Gareth was been dealing with crazy people and weather even before he launched Hot Topic, and back in 2006 he got his hands on on Ring's predictions for rainfall and sunshine hours in each of the major centres. Gareth compared those predictions to the actual values and to the long term average kept by NIWA.

There's a bunch of stuff that can be done with that data (which Gareth kindly made available for anyone to download), but the most important thing (and this holds for almost any statistical analysis) is to take a look at it. Here's what I came up with for the rainfall data using the rather wonderful ggplot2 and inkscape :

You can click on the graphic above to get a bigger version. In these graphs the long term average for each observation is the black line, Ring's predictions if the blue circle and the actual rainfall for that month in that city is the green circle.

Now, let's test the accuracy of Ring's predictions, but against what? Statistical tests often compare a set of results to what you might expect to get "by chance" but that's not very helpful in this case. For one, it's not clear what the range of possible values should be, and second, comparing the forecasts to numbers picked at random ignores the seasonal effects everyone knows contribute to weather. You don't have to know the position of the moon 18 years ago to know that Auckland is more rainy in July than January. Instead, let's compare Ring's forecasts to the easiest forecast you could ever make - just saying rainfall for a given region in a given month would match the long-term average. If there was anything to Ring's methods he should be able to do better than that. He didn't, on average Ring's prediction was 52.5 mm out from the actual rainfall where as "predicting" the average would have been 37.25 mm out.The long-term average was a better predictor, averaging 15.15 mm closer to the actual rainfall with a confidence interval spanning from -0.4 and 31 mm. We can't (quite) say from that data that Ring's forecasts have less predictive power than the long term average, but there is absolutely no evidence they're any better.

Let's lower the bar a little and forget about how close to the true value Ring's predictions came. Was he at least able to get on the right side of the average? If he predicted a drier August than average was the real value likely to indeed be direr? In this case, if you just tossed a coin 48 times, calling heads drier and tails wetter, you'd expect to be right about half the time. And would have done better than Ring. His prediction was only on the right side of the average 17 times in 48 attempts, about 35% accuracy and significantly worse that you would expect to get from tossing a coin (if you're one of those p-value fetishists p, in this case, is equal to about 0.03) .

So Ken Ring can't predict the weather. That probably doesn't surprise anyone reading my blog or at sciblogs.I wish it surprised me that the uselessness of Ring's forecasts is no barrier to him being taken seriously, selling books and appearing on radio and in newspapers. The host of the show that started by little investigation evidently took some flack for having Ring on (it was a National Radio after all) and his defense amounted to "well, we just let these people talk and you can decide what you think about it". That seems impossibly weak to me, surely the least we should do is ask people who make outrageous claims to show what evidence they have to support those claims. I'm pretty satisfied there is nothing in astrological weather forecasting, but if advocates of the method really want to put it to the test it would be easy. Get someone to provide a year's worth of old weather maps but let only half of them be of the right vintage for their method (about 18 years in Ring's case) and see if the forecasts from the supposedly predictive weather maps are better than the ones from weather maps chosen at random. How likely do you think it is that test will be run?

Labels: pretty data, sci-blogs, science and society, skepticism

1 Comments:

 Myrtle George, at March 6, 2011 at 11:53 PM  

Post a Comment

All content not otherwise marked is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.