Current Alerts 5 currently active, find out more »

Generating forecasts for Beach Report and Yarra Watch

EPA personnel looking at water quality at Port Phillip Bay, Melbourne CBD in the background

An EPA staff member samples beach water. Source: EPA Victoria

How forecasts are generated

Environment Protection Authority Victoria (EPA) forecasts water quality for popular swimming areas along the Yarra River and at beaches around Port Phillip Bay during the summer months.

Forecasts are reported on the Yarra and Bay website and via Twitter (@EPA_Victoria), allowing swimmers to decide whether they swim or not that day. Forecasts rate the water quality as either ‘Good’, ‘Fair’ or ‘Poor’ and are a prediction of bacterial pollution in the water.

This page provides an overview of how forecasts are generated, and the accuracy of forecasts. This information may be particularly useful to organisations that undertake water quality sampling and forecast water quality in other jurisdictions. For further information on the methods below, please contact EPA.

Beach Report forecasts

Beach Report provides forecasts for water quality at 36 beaches around Port Phillip Bay during summer. Forecasts for the same day are issued at 10.00 am and 3:00 pm, and a next-day forecast is also issued daily at 3.00 pm. 

Forecasts are generated using a matrix established in the early 2000’s. The matrix generates forecasts for any time, based on a combination of historical water quality data (measuring enterococci bacteria), past and predicted rainfall, and cloud cover conditions. These variables are given a weighting based on how likely they are to influence water quality (see Table 1).

In the matrix, historical water quality and rainfall from the Australian Bureau of Meteorology (BOM) are given the highest weighting. Cloud cover is given a lower weighting. This information is combined to produce a forecast score, where less than 10 is Good, 10-16 is Fair and greater than 16 is Poor. Good quality water is suitable for swimming, while Fair may not be suitable and Poor is unsuitable.

Rainfall has a significant influence on water quality, because rainfall causes stormwater from the streets and urban areas to flow into the Bay, carrying bacteria-laden pollutants such as faecal matter.

The Beach Report forecasting matrix draws on regional-scale information and so is not entirely site-specific; forecasts can be provided for the whole bay or for regions within the bay, with some allowance for individual site susceptibility to stormwater pollution.

Table 1. Matrix for forecasting microbial water quality

Factors

 Criteria

Score

Weighting Final score 

Microbiological sampling history

1. Beach history (5 year 95th percentile) ^(enterococci/100ml)

Good <201

0

2
Fair 201-400 1
Poor >400  2

Rainfall

2. Rainfall in past 24 hours (based on BOM rainfall forecast)

0-1 mm

0 2
1-5 mm  2
5-10 mm 3
10-15 mm 4
15-25 mm  5

3. Predicted rainfall for the forecast period

0-1 mm

0

3  
1-5 mm 2
5-10 mm  3
10-15 mm  4
15-25 mm  5

4. Cloud cover for the forecast period 

Fine

0  1  
Partly cloudy  1
Cloudy  2
 Forecast score * = sum of final scores

* Forecast number ratings and advice:

  • < 10 = Good  – Good quality water is suitable for swimming
  • 10-16 = Fair – Fair may not be suitable for swimming; check for signs of stormwater pollution
  • > 16 =  Poor – Poor is unsuitable for swimming

^ 95th percentile: The 95th percentile is a statistical measure to show the range in which 95% of data is likely to lie. After the data have been ordered from smallest to largest, the 95th percentile value means at least 95% of the data are less than or equal to this value, and 5% of the data are greater than or equal to this value. 

Supplementary checklist to improve accuracy

In the 2011-12 summer, a supplementary checklist was added to the forecasting process, allowing forecasts generated from the matrix to be further refined by considering additional conditions including:

  • significant rain events over the past week
  • increased river flows
  • occurrence of emergency sewage releases
  • forecasts that transitioned from Poor to Fair before returning to Good after heavy rain.
  • fish deaths
  • pollution reports
  • algal blooms.

The location, spread and severity of these conditions may prompt forecasters to downgrade an otherwise Good or Fair forecast to Poor.

Water sampling to support forecasting

Waters samples are collected on the same day each week during summer at all 36 beaches and analysed for enterococci in a NATA-accredited laboratory. The sampling supports forecasting through providing the ‘microbial sampling history’, and results can be used to retrospectively assess the accuracy of forecasts by comparing the actual water quality with forecast water quality. In addition, resampling triggered by high bacterial results detected in routine weekly sampling can also be used to inform forecasts, especially after storms when water quality can be harder to predict.

How accurate are the Beach Report forecasts?

EPA has measured the accuracy of forecasts twice – once in 2010-11 and more recently in 2016-17 summer seasons.

The accuracy of Beach Report forecasts was measured by comparing daily forecasts to weekly bacterial water quality sampling results at beaches. Note that weekly sampling was conducted on Tuesdays in 2016-17 and Mondays in 2010-11. As such, the accuracy test compares the weekly sample to the corresponding forecast on the same day. Because of this, the accuracy assessments below cannot fully represent the accuracy of all forecasts over the summer period, however it gives a good indication of how well the forecasts predicted actual water quality (see Table 2 and 3 below).

Table 2. Results of accuracy assessments for 2016-17 and 2010-11 reporting seasons – metrics for overall accuracy. Note that the supplementary checklist was introduced after the 2010-11 season to help improve the accuracy of forecasts.

Year of accuracy measurement

Overall accuracy results 

Metric

What does it mean?

2016-17

Appropriate advice

98%

Appropriate advice means that the forecast advice is appropriate to protect the health of swimmers, even if the forecast may not be correct. This is the most common measure of forecast accuracy used by forecast services worldwide.

In 2016-17, 98 per cent of forecasts provided appropriate advice about whether it’s safe to swim. This means that 490 out of the 499 forecasts issued on Tuesdays in the 2016-17 summer provided appropriate advice.

For example, when actual water quality was Good, EPA issued either Good or Fair forecasts; and when water quality was Poor, EPA issued either Poor or Fair forecasts.

Fair forecast ratings advise the beach user that water quality may not be suitable, to check for signs of pollution before swimming. This rating is therefore appropriate for either Good or Poor water quality, as caution is advised.

2010-11

Appropriate advice 

86%

Appropriate advice was given 86 per cent of the time. This means that 464 out of the 540 forecasts issued on Mondays in the 2010-11 summer provided appropriate advice to protect public health.

2016-17

Forecast accuracy

78%

78 per cent of forecasts correctly predicted the actual water quality. This means that 388 out of the 499 forecasts issued on Tuesdays in the 2016-17 summer were correct (i.e. Good forecasts for days with actual Good water quality, Fair forecast for days with actual Fair water quality days, and Poor forecasts for days with actual Poor water quality).

2010-11

Forecast accuracy

 40%

The overall forecast accuracy for the 2010-11 season was 40 per cent.  This means that 218 out of the 540 forecasts issued on Mondays in the 2010-11 summer were correct (i.e. Good forecasts for actually Good water quality, Fair forecast for actually Fair water quality days, and Poor forecasts for actually Poor water quality).

Table 3. Results of accuracy assessments for 2016-17 and 2010-11 reporting seasons – metrics for missed and false alarms.

Year of accuracy measurement

Missed alarms and false alarms

Metric

What does it mean?

2016-17

Missed alarms on poor water quality days

<2%

 

A missed alarm occurs when the forecast is Good, but the actual water quality is found to be Poor. This is the main type of incorrect forecast we aim to reduce, as it may result in people being exposed to Poor water quality and an increased risk of illness.

Missed alarms were issued for eight out of 499 forecasts issued on Tuesdays over summer, or <2 per cent of total forecasts issued.  Of the 19 Poor water quality sample results on Tuesdays we issued Good forecasts for eight of them. However,  these eight missed alarms related to only one or two beaches at a time; the remaining 34-35 beaches on those days were provided with appropriate advice.

In 2016-17 many sampling days (Tuesdays) occurred several days after heavy rain had stopped and so the forecasting matrix and checklist may have predicted water quality returning to Good earlier than it did in reality. This reflects the difficulties of forecasting several days after heavy rain events when bacterial water quality can be highly variable due to:

  • continuing stormwater runoff, or
  • sediment from stormwater pollution containing bacteria that can be resuspended in water by strong onshore winds.

To check the accuracy of forecasts during and after heavy rain, we sampled water quality during rain on Friday 20 January and during the weekend of 21-22 January. This meant that we could check our accuracy during and straight after heavy rain, and adjust the forecast accordingly. On this occasion, we issued no missed alarms.

2010-11

Missed alarms on poor water quality days  

<2%

Missed alarms were issued for seven out of 540 forecasts issued on Tuesdays over summer, or <2 per cent of total forecasts issued.

Of the 64 Poor water quality sample results on Mondays over the 2010-11 summer (out of 540 samples in total), we issued Good forecasts for seven of them.  These mostly occurred during periods after heavy rain where the matrix returned water forecasts to Good too early.

2016-17

False alarms on actual Good water quality days

<1%

Out of the 499 forecasts issued on Tuesdays over the 2016-17 summer, only one Poor forecast was issued for a site with actual Good water quality.

These false alarms may have economic and social impacts if issued often, but do not impact human health.

2010-11

False alarms on actual Good water quality days

13%

Out of the 540 forecasts issued on Mondays over the 2010-11 summer, 69 Poor forecasts were issued for a site with actual Good water quality, or 13% of total forecasts issued.

Yarra Watch Forecasts

Forecast models

Yarra Watch issues twice-daily forecasts (10:00 am and 3:00 pm), including a next-day forecast at 3:00 pm. Yarra Watch forecasts and monitors microbial water quality at four swimming sites along the Yarra River during summer. The small number of sites has made the development of site-specific statistical models easier than for Beach Report.

The current site-specific models for Yarra Watch were developed in 2013 and are statistical models generated for each site. They are based on the relationship between historical microbial water quality and a rain gauge for each site. For further information on the methods, please contact EPA.

How accurate were Yarra Watch forecasts in 2016-17?

The accuracy of Yarra Watch forecasts was measured by comparing morning forecasts (10:00 am) to weekly bacterial water quality sampling results at each Yarra River swimming spot. Note that forecasts are produced daily, but water sampling only occurs once a week on a Wednesday. So, the accuracy test only compares the Wednesday sample to the corresponding forecast on the same day. Because of this, the accuracy assessment below cannot fully represent the accuracy of all forecasts over the summer period, but gives a general indication of forecast accuracy.

Most Yarra Watch forecasts provided appropriate advice to protect public health (Table 4 and 5). Most days of Poor water quality were issued with forecasts advising against swimming or to check for signs of pollution before entering the water.

Table 4. Results of accuracy assessments for 2016-17 reporting season – metrics for overall accuracy.

Overall accuracy results

Metric

What does it mean?

Appropriate advice

93%

This is the most common measure of forecast accuracy used by other forecast services worldwide. 93 per cent of forecasts provided appropriate advice about whether it’s safe to swim. In other words, 52 out of the 56 forecasts issued on Wednesdays this summer provided appropriate advice.

For example, when actual water quality was Good EPA issued either Good or Fair forecasts. When water quality was Poor, EPA issued either Poor or Fair forecasts.

Fair forecast ratings advise the swimmer that water quality may not be suitable, to check for signs of pollution before swimming. This rating is therefore appropriate for either Good or Poor water quality, as caution is advised.

Forecast accuracy

50%

50 per cent of forecasts correctly predicted the actual water quality. In other words 28 out of the 56 forecasts issued on Wednesdays this summer were correct.

This means that 50 per cent of forecasts accurately predicted the actual water quality (i.e. Good forecasts for Good water quality, Fair or Fair water quality days, and Poor forecasts for Poor water quality).

Table 5.Results of accuracy assessments for 2016-17 reporting season – metrics for missed and false alarms.

Missed alarms and false alarms

Metric

What does it mean? 

Missed alarms on poor water quality days

0%

There were no Good forecasts issued on days with actual Poor water quality.

False alarms on actual Good water quality days

7%

Out of the 56 forecasts issued on Wednesdays over summer, only four Poor forecasts were issued for sites with Good water quality. This means that on some occasions we advised people not to swim when actual water quality was Good. These types of alarms may have economic and social impacts if issued often.

Accuracy of Yarra Watch forecasting in 2013-14

An accuracy assessment of Yarra Watch forecasts was conducted in 2013-14 using a different calculation of accuracy to that used in 2016-17. The percentage of forecasts correctly predicting actual water quality for 10:00 am, 3:00 pm and next day forecasts was averaged for each site (based on weekly samples). The assessment did not calculate ‘appropriate advice’. Results are shown in Table 6 below.

Table 6. Forecast accuracy based on comparison of forecasts with actual water quality.

 

Average forecast accuracy over 2013-14 season 

Healesville

64%

Kew

67%

Launching Place

67%

Warrandyte

87%

The accuracy of the site-specific Yarra Watch forecasts ranged between 64-87 per cent (Table 6). Forecast accuracy was considerably better for Warrandyte than other sites as the microbial water quality there is more stable.

Conclusions about forecast-accuracy assessments

Forecast accuracy assessments are used to monitor the appropriateness of forecasts, to help us improve the advice we give about the suitability of water quality for swimming. Accuracy results for 2016-17 indicate that our forecasts generally provide appropriate advice on water quality and that we rarely issue missed-alarms or false-alarms.

The main focus of accuracy assessments is to inform any improvements to our forecasting. It is difficult to make clear comparisons of accuracy between summers as the accuracy results are variable and are influenced by a range of environmental factors other than forecast skill. For example, forecast accuracy can vary between summers due to:

  • forecasting being easier when there is a higher percentage of actual Good water quality days during weekly sampling, as opposed to forecasting during more variable water quality. This variability can be caused by:
    • fluctuating rainfall over a summer
    • the fact that weekly sampling only represents a snapshot of water quality over the summer. For example, in 2016-17 sampling didn’t always capture water quality conditions during rain as wet weather tended occurred during weekends (note sampling was conducted on Tuesdays).
  • changes in forecasting and sampling methods between summers
  • changes in forecast teams between summers (e.g. there may be lower accuracy when there is a shift from experienced to new forecasters in a team).

Due to potential variability in forecast accuracy, as shown between summers of 2010-11, 2013-14 and 2016-17, EPA will continue to try to improve forecast accuracy. This may include the development and trial of site-specific statistical models and hydrodynamic modeling for Beach Report. Further, the addition of new variables like wind, tides and rain gauges closer to Beach Report and Yarra Watch sites would also likely increase forecast accuracy. For example, for Beach Report one limitation observed in the 2016-17 summer is the influence of strong onshore winds in the week (or even month) following storms, which can potentially elevate microbial levels through sediment re-suspension. As this is highly variable, more site-specific information may help improve forecasting accuracy.

Program Partners

Department of Environment and Primary Industries Environmental Protection Agency Melbourne Water