Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules
In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules, especially regarding quality of the forecasted dependency structures. Additionally, we propose scoring rules based on the copula, which uniquely describes the dependency structure for every probability distribution with continuous marginal distributions. Efficient estimation of the considered scoring rules and evaluation methods such as the Diebold-Mariano test are discussed. In detailed simulation studies, we compare the performance of the renowned scoring rules and the ones we propose. Besides extended synthetic studies based on recently published results we also consider a real data example. We find that the energy score, which is probably the most widely used multivariate scoring rule, performs comparably well in detecting forecast errors, also regarding dependencies. This contradicts other studies. The results also show that a proposed copula score provides very strong distinction between models with correct and incorrect dependency structure. We close with a comprehensive discussion on the proposed methodology.
Code (0)
등록된 구현이 없습니다.
Tasks
scoring ruleMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting
Multivariate Gaussian (MVG) distributions are central to modeling correlated continuous variables in probabilistic forecasting. Neural forecasting models typically parameterize the mean vector and covariance matrix of th…
Probabilistic Time Series Forecastingscoring ruleTime Series ForecastingUncertainty QuantificationRegions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts
Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficien…
scoring ruleTime SeriesTime Series ForecastingProbabilistic multivariate electricity price forecasting using implicit generative ensemble post-processing
The reliable estimation of forecast uncertainties is crucial for risk-sensitive optimal decision making. In this paper, we propose implicit generative ensemble post-processing, a novel framework for multivariate probabil…
Decision MakingDeep Generative Transformers for Probabilistic Time Series and Spatiotemporal Forecasting
Reliable uncertainty quantification is paramount for forecasting multivariate time series and spatiotemporal data. While Transformer architectures excel at sequence modeling, current probabilistic approaches typically re…
Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature
Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a…
Probabilistic Time Series ForecastingTime SeriesTime Series Forecasting