A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes. Müller, W., A., Appenzeller, C., Doblas-Reyes, F., J., & Liniger, M., A. ![A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes [pdf]](https://bibbase.org/img/filetypes/pdf.svg "pdf")
Paper abstract bibtex The ranked probability skill score (RPSS) is a widely used measure to quantify the skill of ensemble forecasts. The underlying score is defined by the quadratic norm and is comparable to the mean squared error (mse) but it is applied in probability space. It is sensitive to the shape and the shift of the predicted probability distributions. However, the RPSS shows a negative bias for ensemble systems with small en-semble size, as recently shown. Here, two strategies are explored to tackle this flaw of the RPSS. First, the RPSS is examined for different norms L (RPSS L). It is shown that the RPSS Lϭ1 based on the absolute rather than the squared difference between forecasted and observed cumulative probability distribution is unbiased; RPSS L defined with higher-order norms show a negative bias. However, the RPSS Lϭ1 is not strictly proper in a statistical sense. A second approach is then investigated, which is based on the quadratic norm but with sampling errors in climatological probabilities considered in the reference forecasts. This technique is based on strictly proper scores and results in an unbiased skill score, which is denoted as the debiased ranked probability skill score (RPSS D) hereafter. Both newly defined skill scores are independent of the ensemble size, whereas the associated confidence intervals are a function of the ensemble size and the number of forecasts. The RPSS Lϭ1 and the RPSS D are then applied to the winter mean [December–January–February (DJF)] near-surface temperature predictions of the ECMWF Seasonal Forecast System 2. The overall structures of the RPSS Lϭ1 and the RPSS D are more consistent and largely independent of the ensemble size, unlike the RPSS Lϭ2 . Furthermore, the minimum ensemble size required to predict a climate anomaly given a known signal-to-noise ratio is determined by employing the new skill scores. For a hypothetical setup comparable to the ECMWF hindcast system (40 members and 15 hindcast years), statistically significant skill scores were only found for a signal-to-noise ratio larger than ϳ0.3.
@article{
title = {A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes},
type = {article},
id = {d9b1be3d-fcbc-35ed-874a-f553cdc2cf73},
created = {2015-12-23T14:16:26.000Z},
file_attached = {true},
profile_id = {feefc994-8ca3-3894-85b7-6a253344aebe},
group_id = {bbe77cbc-7d0d-3a6b-a71b-f666abca3e24},
last_modified = {2015-12-23T14:16:26.000Z},
read = {false},
starred = {false},
authored = {false},
confirmed = {false},
hidden = {false},
abstract = {The ranked probability skill score (RPSS) is a widely used measure to quantify the skill of ensemble forecasts. The underlying score is defined by the quadratic norm and is comparable to the mean squared error (mse) but it is applied in probability space. It is sensitive to the shape and the shift of the predicted probability distributions. However, the RPSS shows a negative bias for ensemble systems with small en-semble size, as recently shown. Here, two strategies are explored to tackle this flaw of the RPSS. First, the RPSS is examined for different norms L (RPSS L). It is shown that the RPSS Lϭ1 based on the absolute rather than the squared difference between forecasted and observed cumulative probability distribution is unbiased; RPSS L defined with higher-order norms show a negative bias. However, the RPSS Lϭ1 is not strictly proper in a statistical sense. A second approach is then investigated, which is based on the quadratic norm but with sampling errors in climatological probabilities considered in the reference forecasts. This technique is based on strictly proper scores and results in an unbiased skill score, which is denoted as the debiased ranked probability skill score (RPSS D) hereafter. Both newly defined skill scores are independent of the ensemble size, whereas the associated confidence intervals are a function of the ensemble size and the number of forecasts. The RPSS Lϭ1 and the RPSS D are then applied to the winter mean [December–January–February (DJF)] near-surface temperature predictions of the ECMWF Seasonal Forecast System 2. The overall structures of the RPSS Lϭ1 and the RPSS D are more consistent and largely independent of the ensemble size, unlike the RPSS Lϭ2 . Furthermore, the minimum ensemble size required to predict a climate anomaly given a known signal-to-noise ratio is determined by employing the new skill scores. For a hypothetical setup comparable to the ECMWF hindcast system (40 members and 15 hindcast years), statistically significant skill scores were only found for a signal-to-noise ratio larger than ϳ0.3.},
bibtype = {article},
author = {Müller, W A and Appenzeller, C and Doblas-Reyes, F J and Liniger, M A}
}
Downloads: 0
{"_id":"ryajaQBBzejBKDfZ9","bibbaseid":"mller-appenzeller-doblasreyes-liniger-adebiasedrankedprobabilityskillscoretoevaluateprobabilisticensembleforecastswithsmallensemblesizes","downloads":0,"creationDate":"2015-12-23T14:16:20.711Z","title":"A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes","author_short":["Müller, W., A.","Appenzeller, C.","Doblas-Reyes, F., J.","Liniger, M., A."],"year":null,"bibtype":"article","biburl":null,"bibdata":{"title":"A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes","type":"article","id":"d9b1be3d-fcbc-35ed-874a-f553cdc2cf73","created":"2015-12-23T14:16:26.000Z","file_attached":"true","profile_id":"feefc994-8ca3-3894-85b7-6a253344aebe","group_id":"bbe77cbc-7d0d-3a6b-a71b-f666abca3e24","last_modified":"2015-12-23T14:16:26.000Z","read":false,"starred":false,"authored":false,"confirmed":false,"hidden":false,"abstract":"The ranked probability skill score (RPSS) is a widely used measure to quantify the skill of ensemble forecasts. The underlying score is defined by the quadratic norm and is comparable to the mean squared error (mse) but it is applied in probability space. It is sensitive to the shape and the shift of the predicted probability distributions. However, the RPSS shows a negative bias for ensemble systems with small en-semble size, as recently shown. Here, two strategies are explored to tackle this flaw of the RPSS. First, the RPSS is examined for different norms L (RPSS L). It is shown that the RPSS Lϭ1 based on the absolute rather than the squared difference between forecasted and observed cumulative probability distribution is unbiased; RPSS L defined with higher-order norms show a negative bias. However, the RPSS Lϭ1 is not strictly proper in a statistical sense. A second approach is then investigated, which is based on the quadratic norm but with sampling errors in climatological probabilities considered in the reference forecasts. This technique is based on strictly proper scores and results in an unbiased skill score, which is denoted as the debiased ranked probability skill score (RPSS D) hereafter. Both newly defined skill scores are independent of the ensemble size, whereas the associated confidence intervals are a function of the ensemble size and the number of forecasts. The RPSS Lϭ1 and the RPSS D are then applied to the winter mean [December–January–February (DJF)] near-surface temperature predictions of the ECMWF Seasonal Forecast System 2. The overall structures of the RPSS Lϭ1 and the RPSS D are more consistent and largely independent of the ensemble size, unlike the RPSS Lϭ2 . Furthermore, the minimum ensemble size required to predict a climate anomaly given a known signal-to-noise ratio is determined by employing the new skill scores. For a hypothetical setup comparable to the ECMWF hindcast system (40 members and 15 hindcast years), statistically significant skill scores were only found for a signal-to-noise ratio larger than ϳ0.3.","bibtype":"article","author":"Müller, W A and Appenzeller, C and Doblas-Reyes, F J and Liniger, M A","bibtex":"@article{\n title = {A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes},\n type = {article},\n id = {d9b1be3d-fcbc-35ed-874a-f553cdc2cf73},\n created = {2015-12-23T14:16:26.000Z},\n file_attached = {true},\n profile_id = {feefc994-8ca3-3894-85b7-6a253344aebe},\n group_id = {bbe77cbc-7d0d-3a6b-a71b-f666abca3e24},\n last_modified = {2015-12-23T14:16:26.000Z},\n read = {false},\n starred = {false},\n authored = {false},\n confirmed = {false},\n hidden = {false},\n abstract = {The ranked probability skill score (RPSS) is a widely used measure to quantify the skill of ensemble forecasts. The underlying score is defined by the quadratic norm and is comparable to the mean squared error (mse) but it is applied in probability space. It is sensitive to the shape and the shift of the predicted probability distributions. However, the RPSS shows a negative bias for ensemble systems with small en-semble size, as recently shown. Here, two strategies are explored to tackle this flaw of the RPSS. First, the RPSS is examined for different norms L (RPSS L). It is shown that the RPSS Lϭ1 based on the absolute rather than the squared difference between forecasted and observed cumulative probability distribution is unbiased; RPSS L defined with higher-order norms show a negative bias. However, the RPSS Lϭ1 is not strictly proper in a statistical sense. A second approach is then investigated, which is based on the quadratic norm but with sampling errors in climatological probabilities considered in the reference forecasts. This technique is based on strictly proper scores and results in an unbiased skill score, which is denoted as the debiased ranked probability skill score (RPSS D) hereafter. Both newly defined skill scores are independent of the ensemble size, whereas the associated confidence intervals are a function of the ensemble size and the number of forecasts. The RPSS Lϭ1 and the RPSS D are then applied to the winter mean [December–January–February (DJF)] near-surface temperature predictions of the ECMWF Seasonal Forecast System 2. The overall structures of the RPSS Lϭ1 and the RPSS D are more consistent and largely independent of the ensemble size, unlike the RPSS Lϭ2 . Furthermore, the minimum ensemble size required to predict a climate anomaly given a known signal-to-noise ratio is determined by employing the new skill scores. For a hypothetical setup comparable to the ECMWF hindcast system (40 members and 15 hindcast years), statistically significant skill scores were only found for a signal-to-noise ratio larger than ϳ0.3.},\n bibtype = {article},\n author = {Müller, W A and Appenzeller, C and Doblas-Reyes, F J and Liniger, M A}\n}","author_short":["Müller, W., A.","Appenzeller, C.","Doblas-Reyes, F., J.","Liniger, M., A."],"urls":{"Paper":"http://bibbase.org/service/mendeley/feefc994-8ca3-3894-85b7-6a253344aebe/file/07f1218c-b3b7-6f6f-761d-cd1bd1ff8f96/A_Debiased_Ranked_Probability_Skill_Score_to_Evaluate_Probabilistic_Ensemble_Forecasts_with_Small_Ensemble_Sizes.pdf.pdf"},"bibbaseid":"mller-appenzeller-doblasreyes-liniger-adebiasedrankedprobabilityskillscoretoevaluateprobabilisticensembleforecastswithsmallensemblesizes","role":"author","downloads":0},"search_terms":["debiased","ranked","probability","skill","score","evaluate","probabilistic","ensemble","forecasts","small","ensemble","sizes","müller","appenzeller","doblas-reyes","liniger"],"keywords":[],"authorIDs":[]}