The Dominion Post in Wellington has run a secret accuracy trial over 30 days comparing forecasts from WeatherWatch.co.nz, MetService and NIWA.  Both NIWA & MetService are based in Wellington, have multi millions in tax funding, access to all New Zealand data and dozens of meteorologists & scientists – yet WeatherWatch.co.nz has matched them accuracy-wise in this independent trial by The Dominion Post/Fairfax.  

 Full details – and reactions from WeatherWatch.co.nz, Metservice and NIWA – below…

 

---

 

Taking a rain check – by Olivia Wannan (from Saturday)

When it comes to weather forecasts, it can pay to shop around.

To check if the washing will get wet, it might help to stop by the National Institute for Water and Atmospheric Research’s forecasts – whereas to discover how hot the day will be it could be best to try the MetService website, and how many blankets the bed requires overnight may be a WeatherWatch question.

The Dominion Post collated 30 days of weather forecasts from the three weather agencies, which all offer their predictions online. MetService and WeatherWatch also have smartphone apps, most of which are free.

The website forecasts for the three agencies – at two and four days ahead – were then compared for their accuracy to recorded weather information. Predictions for rainfall, maximum and minimum temperature were cross-checked.

For their short-range forecasts, all the agencies correctly predicted whether it would rain about two out of every three days.

But when it came to the most accurate, Niwa’s weather forecasts pipped the field, with a few more of their predictions matching the eventuating weather than rivals MetService and WeatherWatch.

Niwa’s weather graphics, launched online last year, are also slightly more conservative, offering only a 4 1/2-day outlook to WeatherWatch’s five and MetService’s nine days in advance.

Perhaps surprisingly, MetService’s long-range forecasts – at six days out – were more accurate than their short-range ones, with their computer- modelled forecasts picking the eventuating rain or shine on five out of every six days.

However, forecasting a rainy day appears to be the right call in the mid-June to mid-July period, with only eight dry days in the 30-day period.

Predicting the daily temperature high appears to be an easier task than picking the low. The three weather services are within one degree of the recorded maximum temperature for about two days out of three.

Here, MetService takes the crown for accuracy in the short- term, getting the day’s high within a degree more than 80 per cent of the time. Highlighting the challenges around forecasting, both Niwa’s and WeatherWatch’s four-day outlooks were ever-so- slightly more accurate than their two-day temperature outlooks.

Predicting the minimum temperature proved elusive, with all agencies’ estimates matching the recorded figures about half the time. But WeatherWatch and Niwa just edge out MetService in the accuracy stakes for estimating the day’s low. Forecasts for wind direction and speed vary in the details they give between MetService, Niwa and WeatherWatch and so were not compared.

Victoria University climate scientist and former forecaster James Renwick says there are two factors that can lead to vastly different weather predictions – the first is data about the climate right now.

“More effort goes into that step than anything else, because the closer you can get to the truth at the beginning of your forecast model, the closer to the truth the answer’s going to be,” he says.

“There’s a massive global infrastructure behind all this and billions of dollars are spent just observing the weather every day to get these starting conditions right.”

This data is then plugged into advanced computer models, which replicate how the atmosphere and the heat within it move around. There are a number of models available – and each might spit out slightly different results from the other.

Some agencies, such as Niwa, stick to one particular model, while others, including MetService, make their predictions after looking across several, Renwick says.

“Forecasting services then run these models dozens of times every day, changing the starting points just a little bit just to see how quickly the outcomes change.”

That’s why many forecasters talk in percentages about tomorrow’s weather, though the predictions are then simplified for public consumption.

Once the computers are done crunching the numbers, meteorologists then have the weather maps with their highs, lows and fronts, he says.

“That doesn’t mean you can accurately forecast rain will fall in Wellington. When you get down to really small scales, like individual towns in New Zealand, a lot of other factors come into play – like how strong the front is, how the mountains interact with the winds. We’re talking [accuracy] of two, three, four days max.”

A few decades ago, Kiwi meteorologists really struggled to get good information to plug into the models, with weather instruments so few and far between, Renwick says. “Twenty years ago, weather forecasts for the southern hemisphere were noticeably worse than they were in the northern hemisphere. But there’s been a lot of development since in satellite observations.

“Just because of chaotic effects, you really can’t get beyond about 10 days. Sometimes when there’s lots going on, the predictability may only last for four or five days.

“Probably the hardest thing for weather forecasters is when there’s nothing happening – not much wind around, but still the potential for a thunderstorm and localised flooding.”

In his old forecasting days, Renwick admits he took a slightly pessimistic take on what the models were telling him – and suspects many of today’s meteorologists do as well. “If there’s uncertainty, it’s better to say it’s going to rain and then have a fine day than to disappoint people.”

THE REACTION

MetService – forecaster John Law:

The incredibly complex nature of weather means it’s very difficult for even the most skilled forecasters to get it 100 per cent accurate, particularly as you get further out in time. But our team of meteorologists work 24/7 to make them as accurate as possible. They are supported by constantly improving computer models from MetService’s forecast research team – we use a selection of the best global models as well as our own local models – and by the observation team’s hard work maintaining the network of weather stations and radars across New Zealand, so we get the most accurate information about what is happening right now.

There will always be areas of improvement to work on with weather forecasting, even for the most skilled meteorologists.

MetService verifies urban forecasts across 41 locations, for false alarms – rain being forecast and not occurring – and “probability of detection” – that rain was forecast and it happened.

In July 2014, false alarms were 28 per cent across the country, and probability of detection was 82 per cent.

Niwa – chief atmosphere, natural hazards and energy scientist Murray Poulter:

Niwa has been involved in weather forecasting for many years. It has the largest team of climate and weather scientists in New Zealand, a national climate station network, the national climate database and one of the most powerful supercomputers of its kind running world-class weather forecasting models over our complex terrain. Niwa uses sophisticated models run on our supercomputer producing “real time” highly detailed and precise localised forecasts.

Forecasting is a major global challenge, involving the atmosphere well beyond our shores and relies on dedicated collaborations with overseas agencies. The recent “Deepwave” experiment involving 100 international scientists and two research aircraft in the South Island is a good example of a collaboration aimed at further improving forecast accuracy.

Our forecasts will continue to improve as we invest in the necessary research and computing infrastructure, allowing us to make finer scale forecasts that are appropriate to New Zealand’s productive sectors.

WeatherWatch.co.nz – head analyst Philip Duncan:

WeatherWatch, with just two full-time forecasters, has managed to be as accurate – and in some cases more accurate – than both Niwa and MetService.

They have dozens of meteorologists and scientists and millions of dollars, including money from government funding.

This is huge for us as, unlike Niwa and MetService, we aren’t based in Wellington – they should have a much bigger advantage.

We don’t receive any funding from the government, no access to New Zealand data, yet here we are pretty much as accurate as them thanks to being fiercely passionate about accuracy and also thanks to the US government, who unlike the New Zealand government, believes weather data should be free for public good.

These results are fantastic for WeatherWatch.co.nz – but we are very motivated to get our accuracy up even further in Wellington and will do our best to do that as we head into 2015.

– The Dominion Post